09/01/2025 01:56:09.346 7f7da2697580 [contrail-api] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/01/2025 01:56:09.346 7f7da2697580 [contrail-api] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/01/2025 01:56:09.346 7f7da2697580 [contrail-api] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/01/2025 01:56:09.346 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to zookeeper on 10.0.0.254:2181 09/01/2025 01:56:09.347 7f7da2697580 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 01:56:09.347 7f7d9ddad900 [contrail-api] [ERROR]: Cannot write http_port 8084 to /tmp/contrail-api.0.http_port 09/01/2025 01:56:09.347 7f7d9ddad900 [contrail-api] [ERROR]: Starting Introspect on HTTP Port 8084 09/01/2025 01:56:09.347 7f7d9ddadcc0 [contrail-api] [INFO]: Processing event[EvStart] in state[none] 09/01/2025 01:56:09.348 7f7d9ddadcc0 [contrail-api] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 01:56:09.348 7f7d9ddadcc0 [contrail-api] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/01/2025 01:56:09.348 7f7d9ddadcc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.348 7f7d9ddadcc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 01:56:09.348 7f7d9ddadcc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.348 7f7d9ddadcc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.348 7f7d9ddadcc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:56:09.348 7f7d9ddadcc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.348 7f7d9ddadcc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.348 7f7d9ddadcc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 09/01/2025 01:56:09.348 7f7d9ddadcc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.353 7f7d9ddadcc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.353 7f7d9ddadcc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:56:09.354 7f7d9ddadcc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.408 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to database on ['10.0.0.254:9041'] 09/01/2025 01:56:09.408 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: VNCCassandra started with driver cql 09/01/2025 01:56:09.408 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: initializing cassandra cluster 09/01/2025 01:56:09.408 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster before create 09/01/2025 01:56:09.409 7f7d9ddadd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.409 7f7d9ddadd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:09.409 7f7d9ddadd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.411 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster created 09/01/2025 01:56:09.456 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 01:56:09.457 7f7d9ddbdea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.457 7f7d9ddbdea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:09.457 7f7d9ddbdea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.462 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 01:56:09.462 7f7d9ddbde00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.462 7f7d9ddbde00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:09.462 7f7d9ddbde00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.481 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:09.943 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'config_db_uuid', created 09/01/2025 01:56:09.984 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:09.998 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:10.177 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'config_db_uuid' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 09/01/2025 01:56:10.193 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:10.650 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:10.727 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'useragent', created 09/01/2025 01:56:10.731 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:11.062 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:11.131 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'useragent' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 09/01/2025 01:56:11.136 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:11.136 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 01:56:11.158 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 01:56:11.480 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:11.689 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_fq_name_table', created 09/01/2025 01:56:11.693 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:11.847 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:11.992 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 01:56:11.998 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:12.383 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:12.575 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_shared_table', created 09/01/2025 01:56:12.579 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:12.752 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:12.904 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 01:56:12.910 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:13.285 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:13.348 7f7d9db69f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:13.349 7f7d9db69f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:13.350 7f7d9db69f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:13.350 7f7d9db69f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:13.350 7f7d9db69f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:13.350 7f7d9db69f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:13.350 7f7d9db69f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:13.351 7f7d9db69f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:13.351 7f7d9db69f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.351 7f7d9db69f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:13.351 7f7d9db69f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.351 7f7d9db69f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.351 7f7d9db69f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:13.351 7f7d9db69f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.486 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'useragent_keyval_table', created 09/01/2025 01:56:13.494 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:13.650 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:13.778 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'useragent_keyval_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 01:56:13.783 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:13.784 7f7d9ddbd0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.784 7f7d9ddbd0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:13.784 7f7d9ddbd0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.870 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.254:5673//'] 09/01/2025 01:56:13.877 7f7d9e882680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.885 7f7d9e882680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:13.885 7f7d9e882680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.886 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection down 09/01/2025 01:56:13.888 7f7d9db69680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.892 7f7d9db69680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:13.893 7f7d9db69680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.898 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:56:13.898 7f7d9db69680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.898 7f7d9db69680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:13.899 7f7d9db69680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.920 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/01/2025 01:56:13.921 7f7d9dbbc860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.923 7f7d9dbbc860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:13.923 7f7d9dbbc860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.928 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:56:13.932 7f7d9dbbc860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.933 7f7d9dbbc860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:13.933 7f7d9dbbc860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.935 7f7d9dbbc2c0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/01/2025 01:56:13.937 7f7d9e882680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.937 7f7d9e882680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:13.940 7f7d9e882680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.943 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config']: Unknown id: global_system_config default-global-system-config 09/01/2025 01:56:13.946 7f7d9dbbc2c0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:56:13.946 7f7d9e882cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.947 7f7d9e882cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:13.947 7f7d9e882cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.990 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-global-vrouter-config']: Unknown id: global_vrouter_config default-global-system-config:default-global-vrouter-config 09/01/2025 01:56:14.033 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain']: Unknown id: domain default-domain 09/01/2025 01:56:14.171 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-fabric']: Unknown id: fabric default-global-system-config:default-fabric 09/01/2025 01:56:14.218 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-policy-management']: Unknown id: policy_management default-policy-management 09/01/2025 01:56:14.231 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-policy-management', 'default-application-policy-set']: Unknown id: application_policy_set default-policy-management:default-application-policy-set 09/01/2025 01:56:14.262 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'ip-fabric']: Unknown id: virtual_network default-domain:default-project:ip-fabric 09/01/2025 01:56:14.333 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric']: Unknown id: routing_instance default-domain:default-project:ip-fabric:ip-fabric 09/01/2025 01:56:14.352 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'ip-fabric', '__default__']: Unknown id: routing_instance default-domain:default-project:ip-fabric:__default__ 09/01/2025 01:56:14.373 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', '__link_local__']: Unknown id: virtual_network default-domain:default-project:__link_local__ 09/01/2025 01:56:14.437 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', '__link_local__', '__link_local__']: Unknown id: routing_instance default-domain:default-project:__link_local__:__link_local__ 09/01/2025 01:56:14.453 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'dci-network']: Unknown id: virtual_network default-domain:default-project:dci-network 09/01/2025 01:56:14.524 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'dci-network', 'dci-network']: Unknown id: routing_instance default-domain:default-project:dci-network:dci-network 09/01/2025 01:56:14.542 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'dci-network', '__default__']: Unknown id: routing_instance default-domain:default-project:dci-network:__default__ 09/01/2025 01:56:14.555 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'system-defined-bottle-request-size-limit']: Unknown id: alarm default-global-system-config:system-defined-bottle-request-size-limit 09/01/2025 01:56:14.569 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network']: Unknown id: routing_instance default-domain:default-project:default-virtual-network:default-virtual-network 09/01/2025 01:56:14.586 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', '__no_rule__']: Unknown id: security_group default-domain:default-project:__no_rule__ 09/01/2025 01:56:14.603 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-discovery-service-assignment']: Unknown id: discovery_service_assignment default-discovery-service-assignment 09/01/2025 01:56:14.613 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-global-qos-config']: Unknown id: global_qos_config default-global-system-config:default-global-qos-config 09/01/2025 01:56:14.627 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'service-chain-flat-ipam']: Unknown id: network_ipam default-domain:default-project:service-chain-flat-ipam 09/01/2025 01:56:14.640 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['label']: Unknown id: tag_type label 09/01/2025 01:56:14.649 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['application']: Unknown id: tag_type application 09/01/2025 01:56:14.662 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['tier']: Unknown id: tag_type tier 09/01/2025 01:56:14.674 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['deployment']: Unknown id: tag_type deployment 09/01/2025 01:56:14.685 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['site']: Unknown id: tag_type site 09/01/2025 01:56:14.696 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['neutron_fwaas']: Unknown id: tag_type neutron_fwaas 09/01/2025 01:56:14.737 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['neutron_tag']: Unknown id: tag_type neutron_tag 09/01/2025 01:56:14.748 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Executing db_resync on api-server instance 10.0.0.254_0 09/01/2025 01:56:14.749 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Contrail Version is(nightly) 09/01/2025 01:56:14.768 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: zk_sync_node data (None) 09/01/2025 01:56:14.768 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DBERESYNC: running dbe-resync 09/01/2025 01:56:14.774 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Error in _create_default_rbac_rule: NoIdError: Unknown id: api_access_list default-global-system-config:default-api-access-list 09/01/2025 01:56:14.776 7f7da2697580 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-api-access-list']: Unknown id: api_access_list default-global-system-config:default-api-access-list 09/01/2025 01:56:14.777 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type fabric len 1 09/01/2025 01:56:14.777 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for fabric 09/01/2025 01:56:14.779 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for fabric 09/01/2025 01:56:14.779 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type network_ipam len 2 09/01/2025 01:56:14.779 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for network_ipam 09/01/2025 01:56:14.782 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for network_ipam 09/01/2025 01:56:14.782 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type tag_type len 7 09/01/2025 01:56:14.782 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for tag_type 09/01/2025 01:56:14.787 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for tag_type 09/01/2025 01:56:14.787 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type project len 1 09/01/2025 01:56:14.787 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for project 09/01/2025 01:56:14.791 7f7d9e882f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'default-application-policy-set']: Unknown id: application_policy_set default-domain:default-project:default-application-policy-set 09/01/2025 01:56:14.906 7f7d9d65b900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.906 7f7d9d65b900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:56:14.906 7f7d9d65b900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.911 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Starting consumer greenlet 09/01/2025 01:56:14.912 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 01:56:14.912 7f7d9d927c20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Starting publisher greenlet 09/01/2025 01:56:14.912 7f7d9d927c20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Publisher connection 09/01/2025 01:56:14.913 7f7da2697580 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Added consumer: amqp_request.cn-jenkins-deploy-platform-ansible-os-3992-1..dummy 09/01/2025 01:56:14.913 7f7da2697580 [contrail-api] [DEBUG]: Add uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 01:56:14.914 7f7d9e882680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.914 7f7d9e882680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ConfigApiWorkerTrace: data = << name = worker-0 worker_id = 0 introspect_port = 8084 admin_port = 8095 introspect_url = http://10.0.0.254:8084 >> 09/01/2025 01:56:14.914 7f7d9e882680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.919 7f7d9d927ae0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Consumer connection established 09/01/2025 01:56:14.920 7f7d9d927c20 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Publisher connection established 09/01/2025 01:56:14.925 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Created consumers ['amqp_request.cn-jenkins-deploy-platform-ansible-os-3992-1..dummy'] 09/01/2025 01:56:14.927 7f7d9db39860 [contrail-api] [DEBUG]: Add uve in the [ObjectProjectTable:ContrailConfigTrace] map 09/01/2025 01:56:14.927 7f7d9db399a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.927 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { fq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3464337027075950676, "uuid_lslong": 13813739778345031294}, "enable": true, "created": "2025-09-01T01:56:14.082739", "last_modified": "2025-09-01T01:56:14.082739", "description": null, "user_visible": true, "creator": null}parent_type : "domain"parent_uuid : "ad8fac03-7fc3-4cff-8182-fd0fd71e19ae"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "3013cc2b-d918-4854-bfb4-4323dbf0567e" } deleted = False >> 09/01/2025 01:56:14.927 7f7d9db399a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.927 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for project 09/01/2025 01:56:14.927 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type namespace len 1 09/01/2025 01:56:14.927 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for namespace 09/01/2025 01:56:14.929 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for namespace 09/01/2025 01:56:14.940 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type security_group len 1 09/01/2025 01:56:14.940 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for security_group 09/01/2025 01:56:14.941 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 01:56:14.944 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for security_group 09/01/2025 01:56:14.945 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type global_vrouter_config len 1 09/01/2025 01:56:14.945 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for global_vrouter_config 09/01/2025 01:56:14.947 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for global_vrouter_config 09/01/2025 01:56:14.947 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type virtual_network len 4 09/01/2025 01:56:14.947 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for virtual_network 09/01/2025 01:56:14.949 7f7d9d927ae0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Consumer connection established 09/01/2025 01:56:14.954 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Created consumers ['amqp_request.cn-jenkins-deploy-platform-ansible-os-3992-1..dummy'] 09/01/2025 01:56:14.968 7f7d9db39ae0 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 09/01/2025 01:56:14.968 7f7d9db39c20 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 09/01/2025 01:56:14.969 7f7d9db39d60 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 09/01/2025 01:56:14.969 7f7d9db39ea0 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 09/01/2025 01:56:14.969 7f7d9d87b040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.969 7f7d9d87b040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:dci-network elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "dci-network"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18130706646470052319, "uuid_lslong": 10722994389669597332}, "enable": true, "created": "2025-09-01T01:56:14.477049", "last_modified": "2025-09-01T01:56:14.955685", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : falseis_shared : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"parent_uuid : "3013cc2b-d918-4854-bfb4-4323dbf0567e"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "fb9d35a2-65a9-49df-94cf-ba95a4c35094"virtual_network_category : "tenant"virtual_network_network_id : 4 } deleted = False >> 09/01/2025 01:56:14.969 7f7d9d87b040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.969 7f7d9d87b040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.969 7f7d9d87b040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:__link_local__ elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "__link_local__"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 6449198762980886726, "uuid_lslong": 9226863535917229124}, "enable": true, "created": "2025-09-01T01:56:14.387281", "last_modified": "2025-09-01T01:56:14.957197", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : falseis_shared : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"parent_uuid : "3013cc2b-d918-4854-bfb4-4323dbf0567e"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "5980281b-095d-48c6-800c-677ff959e444"virtual_network_category : "tenant"virtual_network_network_id : 3 } deleted = False >> 09/01/2025 01:56:14.969 7f7d9d87b040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.969 7f7d9d87b040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.970 7f7d9d87b040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:default-virtual-network elements = { fq_name : ["default-domain", "default-project", "default-virtual-network"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10469706662728712198, "uuid_lslong": 10194150420826761948}, "enable": true, "created": "2025-09-01T01:56:14.137978", "last_modified": "2025-09-01T01:56:14.956773", "description": null, "user_visible": true, "creator": null}is_shared : falseparent_type : "project"parent_uuid : "3013cc2b-d918-4854-bfb4-4323dbf0567e"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "914bdebd-267c-4006-8d78-e5cbfc472adc"virtual_network_network_id : 1 } deleted = False >> 09/01/2025 01:56:14.970 7f7d9d87b040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.970 7f7d9d87b040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.970 7f7d9d87b040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:ip-fabric elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "ip-fabric"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 6774002509724274582, "uuid_lslong": 12359476005429416304}, "enable": true, "created": "2025-09-01T01:56:14.282200", "last_modified": "2025-09-01T01:56:14.956357", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : trueis_shared : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"parent_uuid : "3013cc2b-d918-4854-bfb4-4323dbf0567e"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "5e021764-73b8-4f96-ab85-adf191959170"virtual_network_category : "tenant"virtual_network_network_id : 2 } deleted = False >> 09/01/2025 01:56:14.970 7f7d9d87b040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.970 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for virtual_network 09/01/2025 01:56:14.970 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type application_policy_set len 1 09/01/2025 01:56:14.970 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for application_policy_set 09/01/2025 01:56:14.972 7f7d9db39720 [contrail-api] [DEBUG]: Add uve in the [ObjectApplicationPolicySetTable:ContrailConfigTrace] map 09/01/2025 01:56:14.973 7f7d9db39a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.973 7f7d9db39a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]global_vrouter_config_refs : [{"to": ["default-global-system-config", "default-global-vrouter-config"], "attr": null, "uuid": "824fe8e8-2e04-42cf-81ac-5b76c7f8b538"}]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10731447578935577411, "uuid_lslong": 12718885742145842877}, "enable": true, "created": "2025-09-01T01:56:14.245061", "last_modified": "2025-09-01T01:56:14.252887", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"parent_uuid : "45283741-dfb2-49a3-9a2e-9b94f0da8f2c"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "94edc2b7-242e-4b43-b082-8f31e7622ebd" } deleted = False >> 09/01/2025 01:56:14.973 7f7d9db39a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:14.973 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for application_policy_set 09/01/2025 01:56:14.973 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type global_qos_config len 1 09/01/2025 01:56:14.973 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for global_qos_config 09/01/2025 01:56:14.975 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for global_qos_config 09/01/2025 01:56:14.975 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type domain len 1 09/01/2025 01:56:14.975 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for domain 09/01/2025 01:56:14.981 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for domain 09/01/2025 01:56:14.981 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type alarm len 1 09/01/2025 01:56:14.981 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for alarm 09/01/2025 01:56:14.983 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for alarm 09/01/2025 01:56:14.983 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type global_system_config len 1 09/01/2025 01:56:14.983 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for global_system_config 09/01/2025 01:56:14.984 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for global_system_config 09/01/2025 01:56:14.984 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type routing_instance len 6 09/01/2025 01:56:14.984 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for routing_instance 09/01/2025 01:56:14.988 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for routing_instance 09/01/2025 01:56:14.988 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type discovery_service_assignment len 1 09/01/2025 01:56:14.988 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for discovery_service_assignment 09/01/2025 01:56:14.990 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for discovery_service_assignment 09/01/2025 01:56:14.990 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type policy_management len 1 09/01/2025 01:56:14.990 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for policy_management 09/01/2025 01:56:14.991 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for policy_management 09/01/2025 01:56:14.995 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DBERESYNC: Completed successfully 09/01/2025 01:56:14.997 7f7d9db89040 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Cassandra DB walk completed. 09/01/2025 01:56:15.010 7f7d9db89040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Time elapsed in resyncing db: 0:00:00.261277 09/01/2025 01:56:15.011 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'obj_dict': {'autonomous_system': 64512, 'config_version': '1.0', 'enable_4byte_as': False, 'enable_security_policy_draft': False, 'fq_name': ['default-global-system-config'], 'ibgp_auto_mesh': True, 'id_perms': {'created': '2025-09-01T01:56:13.973072', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:13.973072', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10323580598023415145, 'uuid_mslong': 4103235205771446636}}, 'igmp_enable': False, 'mac_aging_time': 300, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '38f19eae-05e2-496c-8f44-b9db734ce569'}, 'oper': 'CREATE', 'request-id': 'req-a5623381-0be4-409d-af04-d20e412f3616', 'type': 'global_system_config', 'uuid': '38f19eae-05e2-496c-8f44-b9db734ce569'} 09/01/2025 01:56:15.012 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'obj_dict': {'enable_security_logging': True, 'encryption_mode': 'none', 'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'global_max_vmi_flows': 0, 'id_perms': {'created': '2025-09-01T01:56:14.018462', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.018462', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9343943892606170424, 'uuid_mslong': 9389979831992861391}}, 'max_vm_flows_percent': 0, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '824fe8e8-2e04-42cf-81ac-5b76c7f8b538'}, 'oper': 'CREATE', 'request-id': 'req-62ecab01-d971-468e-bb59-abf85462b2c4', 'type': 'global_vrouter_config', 'uuid': '824fe8e8-2e04-42cf-81ac-5b76c7f8b538'} 09/01/2025 01:56:15.013 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain'], 'obj_dict': {'fq_name': ['default-domain'], 'id_perms': {'created': '2025-09-01T01:56:14.048084', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.048084', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9332299622340499886, 'uuid_mslong': 12506403821259541759}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ad8fac03-7fc3-4cff-8182-fd0fd71e19ae'}, 'oper': 'CREATE', 'request-id': 'req-f5afedcb-dc00-428c-8e75-44df32e238d7', 'type': 'domain', 'uuid': 'ad8fac03-7fc3-4cff-8182-fd0fd71e19ae'} 09/01/2025 01:56:15.013 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-namespace'], 'obj_dict': {'_type': 'namespace', '_uuid': None, 'fq_name': ['default-domain', 'default-namespace'], 'id_perms': {'created': '2025-09-01T01:56:14.066514', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.066514', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9634034796331392286, 'uuid_mslong': 271180340954613159}}, 'name': 'default-namespace', 'parent_type': 'domain', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '03c36d15-54ab-45a7-85b2-f79a9d15dd1e'}, 'oper': 'CREATE', 'request-id': 'req-4aba5491-394c-44c7-9b7f-091530df3340', 'type': 'namespace', 'uuid': '03c36d15-54ab-45a7-85b2-f79a9d15dd1e'} 09/01/2025 01:56:15.014 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project'], 'obj_dict': {'_enable_security_policy_draft': False, '_type': 'project', '_uuid': None, 'fq_name': ['default-domain', 'default-project'], 'id_perms': {'created': '2025-09-01T01:56:14.082739', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.082739', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13813739778345031294, 'uuid_mslong': 3464337027075950676}}, 'name': 'default-project', 'parent_type': 'domain', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '3013cc2b-d918-4854-bfb4-4323dbf0567e'}, 'oper': 'CREATE', 'request-id': 'req-4daae4b6-34d8-4104-b21c-178d50da64dc', 'type': 'project', 'uuid': '3013cc2b-d918-4854-bfb4-4323dbf0567e'} 09/01/2025 01:56:15.015 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'obj_dict': {'_type': 'network-ipam', '_uuid': None, 'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'id_perms': {'created': '2025-09-01T01:56:14.099020', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.099020', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10977168598090993601, 'uuid_mslong': 9303220722046160707}}, 'name': 'default-network-ipam', 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '811badf4-c49d-4f43-9856-bcae30c94fc1'}, 'oper': 'CREATE', 'request-id': 'req-9753c309-cf89-446c-a241-fdb1c00f2f95', 'type': 'network_ipam', 'uuid': '811badf4-c49d-4f43-9856-bcae30c94fc1'} 09/01/2025 01:56:15.016 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.016 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { _enable_security_policy_draft : false_type : "project"_uuid : nullfq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3464337027075950676, "uuid_lslong": 13813739778345031294}, "enable": true, "created": "2025-09-01T01:56:14.082739", "last_modified": "2025-09-01T01:56:14.082739", "description": null, "user_visible": true, "creator": null}name : "default-project"parent_type : "domain"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "3013cc2b-d918-4854-bfb4-4323dbf0567e" } deleted = False >> 09/01/2025 01:56:15.016 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.017 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network'], 'obj_dict': {'_fabric_snat': False, '_flood_unknown_unicast': False, '_igmp_enable': False, '_is_provider_network': False, '_layer2_control_word': False, '_mac_aging_time': 300, '_mac_ip_learning_enable': False, '_mac_learning_enabled': False, '_pbb_etree_enable': False, '_pbb_evpn_enable': False, '_port_security_enabled': True, '_type': 'virtual-network', '_uuid': None, '_virtual_network_category': 'tenant', 'fq_name': ['default-domain', 'default-project', 'default-virtual-network'], 'id_perms': {'created': '2025-09-01T01:56:14.137978', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.137978', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10194150420826761948, 'uuid_mslong': 10469706662728712198}}, 'name': 'default-virtual-network', 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '914bdebd-267c-4006-8d78-e5cbfc472adc', 'virtual_network_network_id': 1}, 'oper': 'CREATE', 'request-id': 'req-83db0d6c-abee-43db-a61e-03fdba1e2a08', 'type': 'virtual_network', 'uuid': '914bdebd-267c-4006-8d78-e5cbfc472adc'} 09/01/2025 01:56:15.017 7f7d9db39a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.017 7f7d9db39a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:default-virtual-network elements = { _fabric_snat : false_flood_unknown_unicast : false_igmp_enable : false_is_provider_network : false_layer2_control_word : false_mac_aging_time : 300_mac_ip_learning_enable : false_mac_learning_enabled : false_pbb_etree_enable : false_pbb_evpn_enable : false_port_security_enabled : true_type : "virtual-network"_uuid : null_virtual_network_category : "tenant"fq_name : ["default-domain", "default-project", "default-virtual-network"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10469706662728712198, "uuid_lslong": 10194150420826761948}, "enable": true, "created": "2025-09-01T01:56:14.137978", "last_modified": "2025-09-01T01:56:14.137978", "description": null, "user_visible": true, "creator": null}name : "default-virtual-network"parent_type : "project"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "914bdebd-267c-4006-8d78-e5cbfc472adc"virtual_network_network_id : 1 } deleted = False >> 09/01/2025 01:56:15.017 7f7d9db39a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.019 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-fabric'], 'obj_dict': {'disable_vlan_vn_uniqueness_check': False, 'fabric_enterprise_style': True, 'fq_name': ['default-global-system-config', 'default-fabric'], 'id_perms': {'created': '2025-09-01T01:56:14.196255', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.196255', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11570395168093576477, 'uuid_mslong': 10586389691973256591}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '92ea6955-f3ec-458f-a092-4d1a511d1d1d'}, 'oper': 'CREATE', 'request-id': 'req-84f96b3b-9298-4c06-b9d7-8eaaae126f04', 'type': 'fabric', 'uuid': '92ea6955-f3ec-458f-a092-4d1a511d1d1d'} 09/01/2025 01:56:15.020 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management'], 'obj_dict': {'fq_name': ['default-policy-management'], 'id_perms': {'created': '2025-09-01T01:56:14.226163', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.226163', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11109988394767912748, 'uuid_mslong': 4983293743750859171}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '45283741-dfb2-49a3-9a2e-9b94f0da8f2c'}, 'oper': 'CREATE', 'request-id': 'req-c2873e26-903c-4d68-a425-c47203534f65', 'type': 'policy_management', 'uuid': '45283741-dfb2-49a3-9a2e-9b94f0da8f2c'} 09/01/2025 01:56:15.021 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'obj_dict': {'all_applications': True, 'fq_name': ['default-policy-management', 'default-application-policy-set'], 'id_perms': {'created': '2025-09-01T01:56:14.245061', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.245061', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12718885742145842877, 'uuid_mslong': 10731447578935577411}}, 'parent_type': 'policy-management', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '94edc2b7-242e-4b43-b082-8f31e7622ebd'}, 'oper': 'CREATE', 'request-id': 'req-2ac311ab-adde-4270-8429-0747dcfeb76e', 'type': 'application_policy_set', 'uuid': '94edc2b7-242e-4b43-b082-8f31e7622ebd'} 09/01/2025 01:56:15.022 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'oper': 'UPDATE', 'request-id': 'req-5afefc6b-3c12-4198-b7de-17ecfe4ffab5', 'type': 'application_policy_set', 'uuid': '94edc2b7-242e-4b43-b082-8f31e7622ebd'} 09/01/2025 01:56:15.022 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.022 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10731447578935577411, "uuid_lslong": 12718885742145842877}, "enable": true, "created": "2025-09-01T01:56:14.245061", "last_modified": "2025-09-01T01:56:14.245061", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "94edc2b7-242e-4b43-b082-8f31e7622ebd" } deleted = False >> 09/01/2025 01:56:15.023 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.025 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric'], 'obj_dict': {'fabric_snat': False, 'flood_unknown_unicast': False, 'fq_name': ['default-domain', 'default-project', 'ip-fabric'], 'id_perms': {'created': '2025-09-01T01:56:14.282200', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.282200', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12359476005429416304, 'uuid_mslong': 6774002509724274582}}, 'igmp_enable': False, 'is_provider_network': True, 'layer2_control_word': False, 'mac_aging_time': 300, 'mac_ip_learning_enable': False, 'mac_learning_enabled': False, 'parent_type': 'project', 'pbb_etree_enable': False, 'pbb_evpn_enable': False, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'uuid': '5e021764-73b8-4f96-ab85-adf191959170', 'virtual_network_category': 'tenant', 'virtual_network_network_id': 2}, 'oper': 'CREATE', 'request-id': 'req-94fb9ba0-9cbe-49a6-a4d0-03c57c0a05ea', 'type': 'virtual_network', 'uuid': '5e021764-73b8-4f96-ab85-adf191959170'} 09/01/2025 01:56:15.026 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.026 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]global_vrouter_config_refs : [{"to": ["default-global-system-config", "default-global-vrouter-config"], "attr": null, "uuid": "824fe8e8-2e04-42cf-81ac-5b76c7f8b538"}]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10731447578935577411, "uuid_lslong": 12718885742145842877}, "enable": true, "created": "2025-09-01T01:56:14.245061", "last_modified": "2025-09-01T01:56:14.252887", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"parent_uuid : "45283741-dfb2-49a3-9a2e-9b94f0da8f2c"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "94edc2b7-242e-4b43-b082-8f31e7622ebd" } deleted = False >> 09/01/2025 01:56:15.026 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.026 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.026 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:ip-fabric elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "ip-fabric"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 6774002509724274582, "uuid_lslong": 12359476005429416304}, "enable": true, "created": "2025-09-01T01:56:14.282200", "last_modified": "2025-09-01T01:56:14.282200", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : truelayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "5e021764-73b8-4f96-ab85-adf191959170"virtual_network_category : "tenant"virtual_network_network_id : 2 } deleted = False >> 09/01/2025 01:56:15.026 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.028 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'id_perms': {'created': '2025-09-01T01:56:14.345692', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.345692', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9508301341395236789, 'uuid_mslong': 17266954747766587431}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': 'efa08bdb-dd5f-4027-83f4-45b15f3bb7b5'}, 'oper': 'CREATE', 'request-id': 'req-e1e786a1-6214-4470-ae5a-4552980a6c9a', 'type': 'routing_instance', 'uuid': 'efa08bdb-dd5f-4027-83f4-45b15f3bb7b5'} 09/01/2025 01:56:15.029 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__'], 'id_perms': {'created': '2025-09-01T01:56:14.367149', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.367149', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10863592410275056547, 'uuid_mslong': 6961348811647436280}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': False, 'uuid': '609baddc-83d5-45f8-96c3-3bbd47130ba3'}, 'oper': 'CREATE', 'request-id': 'req-eff6b881-9202-4fbc-9f76-c9832184f298', 'type': 'routing_instance', 'uuid': '609baddc-83d5-45f8-96c3-3bbd47130ba3'} 09/01/2025 01:56:15.029 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__link_local__'], 'obj_dict': {'fabric_snat': False, 'flood_unknown_unicast': False, 'fq_name': ['default-domain', 'default-project', '__link_local__'], 'id_perms': {'created': '2025-09-01T01:56:14.387281', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.387281', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9226863535917229124, 'uuid_mslong': 6449198762980886726}}, 'igmp_enable': False, 'is_provider_network': False, 'layer2_control_word': False, 'mac_aging_time': 300, 'mac_ip_learning_enable': False, 'mac_learning_enabled': False, 'parent_type': 'project', 'pbb_etree_enable': False, 'pbb_evpn_enable': False, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'uuid': '5980281b-095d-48c6-800c-677ff959e444', 'virtual_network_category': 'tenant', 'virtual_network_network_id': 3}, 'oper': 'CREATE', 'request-id': 'req-9a96f29b-a950-48f2-a6f3-2638e7be6aa6', 'type': 'virtual_network', 'uuid': '5980281b-095d-48c6-800c-677ff959e444'} 09/01/2025 01:56:15.030 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.030 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:__link_local__ elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "__link_local__"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 6449198762980886726, "uuid_lslong": 9226863535917229124}, "enable": true, "created": "2025-09-01T01:56:14.387281", "last_modified": "2025-09-01T01:56:14.387281", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "5980281b-095d-48c6-800c-677ff959e444"virtual_network_category : "tenant"virtual_network_network_id : 3 } deleted = False >> 09/01/2025 01:56:15.030 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.032 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__link_local__', '__link_local__'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', '__link_local__', '__link_local__'], 'id_perms': {'created': '2025-09-01T01:56:14.447738', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.447738', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11440918796949822962, 'uuid_mslong': 98191761512615248}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '015cd8e3-08d4-4d50-9ec6-4f077c489df2'}, 'oper': 'CREATE', 'request-id': 'req-f88148bc-d1f7-43cf-add4-299d7b0b670f', 'type': 'routing_instance', 'uuid': '015cd8e3-08d4-4d50-9ec6-4f077c489df2'} 09/01/2025 01:56:15.033 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network'], 'obj_dict': {'fabric_snat': False, 'flood_unknown_unicast': False, 'fq_name': ['default-domain', 'default-project', 'dci-network'], 'id_perms': {'created': '2025-09-01T01:56:14.477049', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.477049', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10722994389669597332, 'uuid_mslong': 18130706646470052319}}, 'igmp_enable': False, 'is_provider_network': False, 'layer2_control_word': False, 'mac_aging_time': 300, 'mac_ip_learning_enable': False, 'mac_learning_enabled': False, 'parent_type': 'project', 'pbb_etree_enable': False, 'pbb_evpn_enable': False, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'uuid': 'fb9d35a2-65a9-49df-94cf-ba95a4c35094', 'virtual_network_category': 'tenant', 'virtual_network_network_id': 4}, 'oper': 'CREATE', 'request-id': 'req-dbb137b6-6929-4794-9f18-4cc1b7ee9fb5', 'type': 'virtual_network', 'uuid': 'fb9d35a2-65a9-49df-94cf-ba95a4c35094'} 09/01/2025 01:56:15.033 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.033 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:dci-network elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "dci-network"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18130706646470052319, "uuid_lslong": 10722994389669597332}, "enable": true, "created": "2025-09-01T01:56:14.477049", "last_modified": "2025-09-01T01:56:14.477049", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "fb9d35a2-65a9-49df-94cf-ba95a4c35094"virtual_network_category : "tenant"virtual_network_network_id : 4 } deleted = False >> 09/01/2025 01:56:15.033 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.035 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'id_perms': {'created': '2025-09-01T01:56:14.535227', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.535227', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10440718937852436364, 'uuid_mslong': 1934287642422035831}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '1ad7f85c-cbfe-4577-90e4-e28e82635f8c'}, 'oper': 'CREATE', 'request-id': 'req-ebab1565-cd55-4a85-852b-c32e20d5de74', 'type': 'routing_instance', 'uuid': '1ad7f85c-cbfe-4577-90e4-e28e82635f8c'} 09/01/2025 01:56:15.035 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'id_perms': {'created': '2025-09-01T01:56:14.550834', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.550834', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12754722512726421240, 'uuid_mslong': 4232360199261602858}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': False, 'uuid': '3abc5d2d-5e7d-442a-b101-e08c43bc6ef8'}, 'oper': 'CREATE', 'request-id': 'req-6a474ebd-8279-4933-bad9-efbdad813a70', 'type': 'routing_instance', 'uuid': '3abc5d2d-5e7d-442a-b101-e08c43bc6ef8'} 09/01/2025 01:56:15.036 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bottle-request-size-limit'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'UveConfigReq.err_info.*.', 'operand2': {'json_value': 'True'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'fq_name': ['default-global-system-config', 'system-defined-bottle-request-size-limit'], 'id_perms': {'created': '2025-09-01T01:56:14.564067', 'description': 'Bottle request size limit exceeded.', 'last_modified': '2025-09-01T01:56:14.564067', 'uuid': {'uuid_lslong': 12015291617959670314, 'uuid_mslong': 16749534564038887252}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e8724cf1-72aa-4754-a6be-e4111936f22a', 'uve_keys': {'uve_key': ['config-node']}}, 'oper': 'CREATE', 'request-id': 'req-197bdd5c-860b-4347-9856-2dbafab70321', 'type': 'alarm', 'uuid': 'e8724cf1-72aa-4754-a6be-e4111936f22a'} 09/01/2025 01:56:15.037 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'id_perms': {'created': '2025-09-01T01:56:14.580692', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.580692', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13819027478312604755, 'uuid_mslong': 15027559500844057551}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': 'd08c9dbf-20db-43cf-bfc7-0c46722f5453'}, 'oper': 'CREATE', 'request-id': 'req-23450467-0182-4349-89e1-a0be7313a515', 'type': 'routing_instance', 'uuid': 'd08c9dbf-20db-43cf-bfc7-0c46722f5453'} 09/01/2025 01:56:15.038 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__'], 'obj_dict': {'configured_security_group_id': 0, 'fq_name': ['default-domain', 'default-project', '__no_rule__'], 'id_perms': {'created': '2025-09-01T01:56:14.596432', 'creator': None, 'description': 'Security group with no rules', 'enable': True, 'last_modified': '2025-09-01T01:56:14.596432', 'permissions': None, 'user_visible': True, 'uuid': {'uuid_lslong': 9323024404773776243, 'uuid_mslong': 11466188586996615179}}, 'parent_type': 'project', 'perms2': {'global_access': 5, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'security_group_entries': {'policy_rule': []}, 'security_group_id': 8000001, 'uuid': '9f2015c4-f792-480b-8162-094cf6d6ab73'}, 'oper': 'CREATE', 'request-id': 'req-a28c6fb3-17de-4459-8586-72a1755211df', 'type': 'security_group', 'uuid': '9f2015c4-f792-480b-8162-094cf6d6ab73'} 09/01/2025 01:56:15.039 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-discovery-service-assignment'], 'obj_dict': {'fq_name': ['default-discovery-service-assignment'], 'id_perms': {'created': '2025-09-01T01:56:14.609443', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.609443', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12140255567381224029, 'uuid_mslong': 11054811205764335333}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '996a9438-8297-46e5-a87a-da1de0375a5d'}, 'oper': 'CREATE', 'request-id': 'req-49f9c3d7-fffc-4034-9e1f-4df0baba5ff1', 'type': 'discovery_service_assignment', 'uuid': '996a9438-8297-46e5-a87a-da1de0375a5d'} 09/01/2025 01:56:15.040 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-qos-config'], 'obj_dict': {'fq_name': ['default-global-system-config', 'default-global-qos-config'], 'id_perms': {'created': '2025-09-01T01:56:14.623486', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.623486', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10526209301056148575, 'uuid_mslong': 4993938077403281}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0011bdf5-d122-4891-9214-9b96ccdd545f'}, 'oper': 'CREATE', 'request-id': 'req-448c4d5f-22f2-422f-b5fd-cdd50cfcc760', 'type': 'global_qos_config', 'uuid': '0011bdf5-d122-4891-9214-9b96ccdd545f'} 09/01/2025 01:56:15.041 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'service-chain-flat-ipam'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'service-chain-flat-ipam'], 'id_perms': {'created': '2025-09-01T01:56:14.636115', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.636115', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10962462681388096447, 'uuid_mslong': 12874818220482906094}}, 'ipam_subnet_method': 'flat-subnet', 'ipam_subnets': {'subnets': [{'addr_from_start': None, 'alloc_unit': 1, 'allocation_pools': [], 'created': None, 'default_gateway': None, 'dhcp_option_list': None, 'dhcp_relay_server': [], 'dns_nameservers': [], 'dns_server_address': None, 'enable_dhcp': True, 'host_routes': None, 'last_modified': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 8}, 'subnet_name': None, 'subnet_uuid': None, 'subscriber_tag': None, 'vlan_tag': None}, {'addr_from_start': None, 'alloc_unit': 1, 'allocation_pools': [], 'created': None, 'default_gateway': None, 'dhcp_option_list': None, 'dhcp_relay_server': [], 'dns_nameservers': [], 'dns_server_address': None, 'enable_dhcp': True, 'host_routes': None, 'last_modified': None, 'subnet': {'ip_prefix': '::ffff', 'ip_prefix_len': 104}, 'subnet_name': None, 'subnet_uuid': None, 'subscriber_tag': None, 'vlan_tag': None}]}, 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b2ac8af5-339c-47ee-9822-7dba239d1bbf'}, 'oper': 'CREATE', 'request-id': 'req-d01d952b-a70c-4802-a197-3f2cc2c011a3', 'type': 'network_ipam', 'uuid': 'b2ac8af5-339c-47ee-9822-7dba239d1bbf'} 09/01/2025 01:56:15.044 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['label'], 'obj_dict': {'display_name': 'label', 'fq_name': ['label'], 'id_perms': {'created': '2025-09-01T01:56:14.645938', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.645938', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 10548922112300320863, 'uuid_mslong': 17890671337592472062}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0000', 'uuid': 'f8486eca-ee76-41fe-9265-4cc542ec885f'}, 'oper': 'CREATE', 'request-id': 'req-c08018ed-1b3f-4f3a-b7d4-312faf4be2d9', 'type': 'tag_type', 'uuid': 'f8486eca-ee76-41fe-9265-4cc542ec885f'} 09/01/2025 01:56:15.046 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['application'], 'obj_dict': {'display_name': 'application', 'fq_name': ['application'], 'id_perms': {'created': '2025-09-01T01:56:14.658480', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.658480', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 10024298895565741416, 'uuid_mslong': 10332739810500952278}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0001', 'uuid': '8f65441c-c972-40d6-8b1d-76bc08895d68'}, 'oper': 'CREATE', 'request-id': 'req-e220f482-eb45-4794-ad18-99b420e99648', 'type': 'tag_type', 'uuid': '8f65441c-c972-40d6-8b1d-76bc08895d68'} 09/01/2025 01:56:15.048 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['tier'], 'obj_dict': {'display_name': 'tier', 'fq_name': ['tier'], 'id_perms': {'created': '2025-09-01T01:56:14.669954', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.669954', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 12129956649137883267, 'uuid_mslong': 3753021543571148685}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0002', 'uuid': '34156935-a2f2-4f8d-a856-434e1cc04883'}, 'oper': 'CREATE', 'request-id': 'req-6ba68bc8-3788-43c3-9101-b00bbf394404', 'type': 'tag_type', 'uuid': '34156935-a2f2-4f8d-a856-434e1cc04883'} 09/01/2025 01:56:15.049 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['deployment'], 'obj_dict': {'display_name': 'deployment', 'fq_name': ['deployment'], 'id_perms': {'created': '2025-09-01T01:56:14.680063', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.680063', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 10498355640641336077, 'uuid_mslong': 7159705940627309417}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0003', 'uuid': '635c629e-8758-4369-91b1-a6d51e25330d'}, 'oper': 'CREATE', 'request-id': 'req-8bad40ef-a03f-4984-a974-811b9a580ba6', 'type': 'tag_type', 'uuid': '635c629e-8758-4369-91b1-a6d51e25330d'} 09/01/2025 01:56:15.051 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['site'], 'obj_dict': {'display_name': 'site', 'fq_name': ['site'], 'id_perms': {'created': '2025-09-01T01:56:14.691786', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.691786', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 9730918305590957860, 'uuid_mslong': 7827681754389368012}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0004', 'uuid': '6ca18315-312f-4ccc-870b-2aa46f185724'}, 'oper': 'CREATE', 'request-id': 'req-6b8702d1-55e0-4f9d-8644-e579151c71b4', 'type': 'tag_type', 'uuid': '6ca18315-312f-4ccc-870b-2aa46f185724'} 09/01/2025 01:56:15.059 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['neutron_fwaas'], 'obj_dict': {'display_name': 'neutron_fwaas', 'fq_name': ['neutron_fwaas'], 'id_perms': {'created': '2025-09-01T01:56:14.731670', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.731670', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 10194193657073549735, 'uuid_mslong': 824257741336888378}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0005', 'uuid': '0b705a0c-73c0-443a-8d79-0d1eb568c1a7'}, 'oper': 'CREATE', 'request-id': 'req-ce4044c4-0f00-4d36-977d-29ab73d7dac5', 'type': 'tag_type', 'uuid': '0b705a0c-73c0-443a-8d79-0d1eb568c1a7'} 09/01/2025 01:56:15.061 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['neutron_tag'], 'obj_dict': {'display_name': 'neutron_tag', 'fq_name': ['neutron_tag'], 'id_perms': {'created': '2025-09-01T01:56:14.744953', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.744953', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 11577008770730221732, 'uuid_mslong': 5204803883642143654}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0006', 'uuid': '483b2d8e-5210-43a6-a0a9-cc23acf4e0a4'}, 'oper': 'CREATE', 'request-id': 'req-372466c2-3a59-4243-9485-ecdb589d3f9d', 'type': 'tag_type', 'uuid': '483b2d8e-5210-43a6-a0a9-cc23acf4e0a4'} 09/01/2025 01:56:15.063 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'oper': 'UPDATE', 'request-id': 'req-2992a118-a245-4e85-b942-60a77c5b6452', 'type': 'network_ipam', 'uuid': '811badf4-c49d-4f43-9856-bcae30c94fc1'} 09/01/2025 01:56:15.066 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-api-access-list'], 'obj_dict': {'api_access_list_entries': {'rbac_rule': [{'rule_field': '', 'rule_object': 'fqname-to-id', 'rule_perms': [{'role_crud': 'CRUD', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'id-to-fqname', 'rule_perms': [{'role_crud': 'CRUD', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'useragent-kv', 'rule_perms': [{'role_crud': 'CRUD', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'documentation', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': '/', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'watch', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}, {'rule_field': 'logical_router_gateway_external', 'rule_object': 'logical-router', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}]}, 'fq_name': ['default-global-system-config', 'default-api-access-list'], 'id_perms': {'created': '2025-09-01T01:56:14.790473', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.790473', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11415208488749037772, 'uuid_mslong': 2586752577299957027}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '23e5fdc3-8bca-4523-9e6a-f7a3e0e8f8cc'}, 'oper': 'CREATE', 'request-id': 'req-a41f32ad-c086-492a-a56c-71d8f717189e', 'type': 'api_access_list', 'uuid': '23e5fdc3-8bca-4523-9e6a-f7a3e0e8f8cc'} 09/01/2025 01:56:15.067 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-application-policy-set'], 'obj_dict': {'all_applications': True, 'fq_name': ['default-domain', 'default-project', 'default-application-policy-set'], 'id_perms': {'created': '2025-09-01T01:56:14.923070', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:14.923070', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10051435290943693447, 'uuid_mslong': 17279550792776762782}}, 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'efcd4be5-21c7-459e-8b7d-df246f862287'}, 'oper': 'CREATE', 'request-id': 'req-91686368-ffce-46af-bb11-3c82bda2e22e', 'type': 'application_policy_set', 'uuid': 'efcd4be5-21c7-459e-8b7d-df246f862287'} 09/01/2025 01:56:15.067 7f7d9dbbc680 [contrail-api] [DEBUG]: Add uve in the [ObjectApplicationPolicySetTable:ContrailConfigTrace] map 09/01/2025 01:56:15.068 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project'], 'oper': 'UPDATE', 'request-id': 'req-e4afffa7-7223-4eb1-bbbc-1d7e96a5c5dc', 'type': 'project', 'uuid': '3013cc2b-d918-4854-bfb4-4323dbf0567e'} 09/01/2025 01:56:15.068 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.068 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:default-application-policy-set elements = { all_applications : truefq_name : ["default-domain", "default-project", "default-application-policy-set"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17279550792776762782, "uuid_lslong": 10051435290943693447}, "enable": true, "created": "2025-09-01T01:56:14.923070", "last_modified": "2025-09-01T01:56:14.923070", "description": null, "user_visible": true, "creator": null}parent_type : "project"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "efcd4be5-21c7-459e-8b7d-df246f862287" } deleted = False >> 09/01/2025 01:56:15.068 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.070 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.070 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { fq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3464337027075950676, "uuid_lslong": 13813739778345031294}, "enable": true, "created": "2025-09-01T01:56:14.082739", "last_modified": "2025-09-01T01:56:15.005318", "description": null, "user_visible": true, "creator": null}parent_type : "domain"parent_uuid : "ad8fac03-7fc3-4cff-8182-fd0fd71e19ae"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}quota : {"defaults": -1}uuid : "3013cc2b-d918-4854-bfb4-4323dbf0567e" } deleted = False >> 09/01/2025 01:56:15.070 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.959 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:16.038 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:56:16.039 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:56:16.039 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:56:16.039 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:56:16.040 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:56:16.040 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:56:16.041 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:56:16.041 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:56:16.041 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:56:16.041 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:56:16.187 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:56:16.188 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:56:16.188 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:56:16.188 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:56:16.193 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:56:16.193 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:56:16.194 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:56:16.198 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:56:16.198 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:56:16.199 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:56:16.199 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:56:16.203 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:56:16.204 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:56:16.204 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:56:16.205 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:56:16.205 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:56:16.205 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:56:16.205 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:56:16.206 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:56:16.206 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:56:16.206 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:56:16.206 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:56:16.210 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:56:16.212 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:56:16.212 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:56:16.316 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'] not found in DB: Unknown id: config_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1. 09/01/2025 01:56:16.329 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = config_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1. url = http://10.0.0.254:8082/config-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_config_node.py remote_ip = 10.0.0.254 body = {"config-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1."], "parent_type": "global-system-config", "config_node_ip_address": "10.0.0.254", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3992-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:16.363 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'], 'obj_dict': {'config_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3992-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'], 'id_perms': {'created': '2025-09-01T01:56:16.354824', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:16.354824', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9760121009163491485, 'uuid_mslong': 4867251844546578506}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '438bf3c3-ca90-484a-8772-ea58b404589d'}, 'oper': 'CREATE', 'request-id': 'req-462e833c-fd61-4ec9-a729-04833a3491c6', 'type': 'config_node', 'uuid': '438bf3c3-ca90-484a-8772-ea58b404589d'} 09/01/2025 01:56:16.366 7f7d9dbbc680 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:ContrailConfigTrace] map 09/01/2025 01:56:16.371 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:16.372 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. elements = { config_node_ip_address : "10.0.0.254"display_name : "cn-jenkins-deploy-platform-ansible-os-3992-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4867251844546578506, "uuid_lslong": 9760121009163491485}, "enable": true, "created": "2025-09-01T01:56:16.354824", "last_modified": "2025-09-01T01:56:16.354824", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "438bf3c3-ca90-484a-8772-ea58b404589d" } deleted = False >> 09/01/2025 01:56:16.372 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:16.960 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:17.350 7f7d9db39680 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:17.353 7f7d9db39680 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:17.353 7f7d9db39680 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:17.353 7f7d9db39680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:17.354 7f7d9db39680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:56:17.354 7f7d9db39680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:17.354 7f7d9db39680 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:17.355 7f7d9db39680 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:17.355 7f7d9db39680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:17.356 7f7d9db39680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:56:17.356 7f7d9db39680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:17.356 7f7d9db39680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:17.356 7f7d9db39680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:56:17.357 7f7d9db39680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:17.457 7f7d9db395e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:17.458 7f7d9db395e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:17.458 7f7d9db395e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:17.961 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:17.990 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:56:17.990 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:56:17.990 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:56:17.990 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:56:17.990 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:56:17.990 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:56:17.990 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:56:17.990 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:56:17.990 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:56:17.990 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:56:17.991 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:56:17.991 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:56:17.991 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:56:17.991 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:56:17.991 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:56:17.991 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:56:17.991 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:56:17.991 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:56:17.991 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:56:17.991 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:56:17.991 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:56:17.991 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:56:17.991 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:56:17.992 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:56:17.992 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:56:17.992 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:56:17.992 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:56:17.992 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:56:17.992 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:56:17.992 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:56:17.992 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:56:17.992 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:56:17.992 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:56:17.992 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:56:17.992 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:56:18.043 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'opencontrail'] not found in DB: Unknown id: service_appliance_set default-global-system-config:opencontrail 09/01/2025 01:56:18.044 7f7d9ddbd180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_appliance_set identifier_name = default-global-system-config:opencontrail url = http://10.0.0.254:8082/service-appliance-sets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.254 body = {"service-appliance-set": {"uuid": null, "fq_name": ["default-global-system-config", "opencontrail"], "parent_type": "global-system-config", "service_appliance_driver": "svc_monitor.services.loadbalancer.drivers.ha_proxy.driver.OpencontrailLoadbalancerDriver", "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 5, "share": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "display_name": "opencontrail"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:18.069 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'opencontrail'], 'obj_dict': {'display_name': 'opencontrail', 'fq_name': ['default-global-system-config', 'opencontrail'], 'id_perms': {'created': '2025-09-01T01:56:18.062333', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:18.062333', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13308450134701355083, 'uuid_mslong': 794445167943437868}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 5, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service_appliance_driver': 'svc_monitor.services.loadbalancer.drivers.ha_proxy.driver.OpencontrailLoadbalancerDriver', 'uuid': '0b066fab-cccb-4e2c-b8b1-1ce2e5dfb04b'}, 'oper': 'CREATE', 'request-id': 'req-a6d06a36-745c-4ff5-b393-c7934e155327', 'type': 'service_appliance_set', 'uuid': '0b066fab-cccb-4e2c-b8b1-1ce2e5dfb04b'} 09/01/2025 01:56:18.228 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:56:18.228 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:56:18.231 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:56:18.232 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:56:18.232 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:56:18.232 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:56:18.232 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:56:18.232 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:56:18.232 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:56:18.232 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:56:18.235 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:56:18.236 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:56:18.236 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:56:18.236 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:56:18.236 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:56:18.236 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:56:18.237 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:56:18.237 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:56:18.237 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:56:18.237 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:56:18.237 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:56:18.238 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:56:18.238 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:56:18.238 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:56:18.238 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:56:18.239 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:56:18.239 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:56:18.239 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:56:18.239 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:56:18.239 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:56:18.239 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:56:18.239 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:56:18.239 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:56:18.239 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:56:18.239 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:56:18.296 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'native'] not found in DB: Unknown id: service_appliance_set default-global-system-config:native 09/01/2025 01:56:18.300 7f7d9ddbd180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_appliance_set identifier_name = default-global-system-config:native url = http://10.0.0.254:8082/service-appliance-sets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.254 body = {"service-appliance-set": {"uuid": null, "fq_name": ["default-global-system-config", "native"], "parent_type": "global-system-config", "service_appliance_driver": "svc_monitor.services.loadbalancer.drivers.native.driver.OpencontrailLoadbalancerDriver", "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 5, "share": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "display_name": "native"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:18.328 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'native'], 'obj_dict': {'display_name': 'native', 'fq_name': ['default-global-system-config', 'native'], 'id_perms': {'created': '2025-09-01T01:56:18.320561', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:18.320561', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12012834018592906030, 'uuid_mslong': 10546301580599119481}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 5, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service_appliance_driver': 'svc_monitor.services.loadbalancer.drivers.native.driver.OpencontrailLoadbalancerDriver', 'uuid': '925bfd69-2dc0-4e79-a6b6-28e4a7f55b2e'}, 'oper': 'CREATE', 'request-id': 'req-1c5a0f9c-4dcc-42d5-9c2b-0fc24ea66297', 'type': 'service_appliance_set', 'uuid': '925bfd69-2dc0-4e79-a6b6-28e4a7f55b2e'} 09/01/2025 01:56:18.431 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_DNS_refs in resource network-ipam: 'virtual_DNS_refs' 09/01/2025 01:56:18.432 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource network-ipam: 'tag_refs' 09/01/2025 01:56:18.432 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_back_refs in resource network-ipam: 'virtual_network_back_refs' 09/01/2025 01:56:18.432 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_router_back_refs in resource network-ipam: 'virtual_router_back_refs' 09/01/2025 01:56:18.432 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource network-ipam: 'instance_ip_back_refs' 09/01/2025 01:56:18.612 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'analyzer-template'] not found in DB: Unknown id: service_template default-domain:analyzer-template 09/01/2025 01:56:18.613 7f7d9ddbd180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:analyzer-template url = http://10.0.0.254:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.254 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "analyzer-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": null, "service_type": "analyzer", "image_name": "analyzer", "service_scaling": false, "interface_type": [{"service_interface_type": "left", "shared_ip": false, "static_route_enable": false}], "flavor": "m1.medium", "ordered_interfaces": true, "service_virtualization_type": "virtual-machine", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "27b8a85e67fe427d9ebc42e690325176", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "analyzer-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:18.639 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'analyzer-template'], 'obj_dict': {'display_name': 'analyzer-template', 'fq_name': ['default-domain', 'analyzer-template'], 'id_perms': {'created': '2025-09-01T01:56:18.629308', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:18.629308', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9723529920275978666, 'uuid_mslong': 7345069075143214980}}, 'parent_type': 'domain', 'parent_uuid': 'ad8fac03-7fc3-4cff-8182-fd0fd71e19ae', 'perms2': {'global_access': 0, 'owner': '27b8a85e67fe427d9ebc42e690325176', 'owner_access': 7, 'share': [{'tenant': 'domain:ad8fac03-7fc3-4cff-8182-fd0fd71e19ae', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': 'm1.medium', 'image_name': 'analyzer', 'instance_data': None, 'interface_type': [{'service_interface_type': 'left', 'shared_ip': False, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': None, 'service_scaling': False, 'service_type': 'analyzer', 'service_virtualization_type': 'virtual-machine', 'version': 1, 'vrouter_instance_type': None}, 'uuid': '65eeed68-729e-4784-86f0-eaf1ecee99aa'}, 'oper': 'CREATE', 'request-id': 'req-0c1085ab-f701-4331-9499-6d1d33c953b7', 'type': 'service_template', 'uuid': '65eeed68-729e-4784-86f0-eaf1ecee99aa'} 09/01/2025 01:56:18.655 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'nat-template'] not found in DB: Unknown id: service_template default-domain:nat-template 09/01/2025 01:56:18.670 7f7d9ddbd180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:nat-template url = http://10.0.0.254:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.254 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "nat-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "in-network-nat", "service_type": "firewall", "image_name": "analyzer", "service_scaling": false, "interface_type": [{"service_interface_type": "management", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "right", "shared_ip": false, "static_route_enable": false}], "flavor": "m1.medium", "ordered_interfaces": true, "service_virtualization_type": "virtual-machine", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "27b8a85e67fe427d9ebc42e690325176", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "nat-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:18.694 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'nat-template'], 'obj_dict': {'display_name': 'nat-template', 'fq_name': ['default-domain', 'nat-template'], 'id_perms': {'created': '2025-09-01T01:56:18.686676', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:18.686676', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13647815383153858180, 'uuid_mslong': 6435737609643904160}}, 'parent_type': 'domain', 'parent_uuid': 'ad8fac03-7fc3-4cff-8182-fd0fd71e19ae', 'perms2': {'global_access': 0, 'owner': '27b8a85e67fe427d9ebc42e690325176', 'owner_access': 7, 'share': [{'tenant': 'domain:ad8fac03-7fc3-4cff-8182-fd0fd71e19ae', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': 'm1.medium', 'image_name': 'analyzer', 'instance_data': None, 'interface_type': [{'service_interface_type': 'management', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'right', 'shared_ip': False, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'in-network-nat', 'service_scaling': False, 'service_type': 'firewall', 'service_virtualization_type': 'virtual-machine', 'version': 1, 'vrouter_instance_type': None}, 'uuid': '59505542-0a72-4ca0-bd66-c7c820ff5684'}, 'oper': 'CREATE', 'request-id': 'req-3f6760a2-2e9f-45ea-a304-2846609f3a75', 'type': 'service_template', 'uuid': '59505542-0a72-4ca0-bd66-c7c820ff5684'} 09/01/2025 01:56:18.744 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'netns-snat-template'] not found in DB: Unknown id: service_template default-domain:netns-snat-template 09/01/2025 01:56:18.748 7f7d9ddbd180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:netns-snat-template url = http://10.0.0.254:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.254 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "netns-snat-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "in-network-nat", "service_type": "source-nat", "image_name": null, "service_scaling": true, "interface_type": [{"service_interface_type": "right", "shared_ip": true, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": true, "static_route_enable": false}], "flavor": null, "ordered_interfaces": true, "service_virtualization_type": "network-namespace", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "27b8a85e67fe427d9ebc42e690325176", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "netns-snat-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:18.765 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'netns-snat-template'], 'obj_dict': {'display_name': 'netns-snat-template', 'fq_name': ['default-domain', 'netns-snat-template'], 'id_perms': {'created': '2025-09-01T01:56:18.760578', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:18.760578', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11759015559310255960, 'uuid_mslong': 15731542269789425242}}, 'parent_type': 'domain', 'parent_uuid': 'ad8fac03-7fc3-4cff-8182-fd0fd71e19ae', 'perms2': {'global_access': 0, 'owner': '27b8a85e67fe427d9ebc42e690325176', 'owner_access': 7, 'share': [{'tenant': 'domain:ad8fac03-7fc3-4cff-8182-fd0fd71e19ae', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': None, 'image_name': None, 'instance_data': None, 'interface_type': [{'service_interface_type': 'right', 'shared_ip': True, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': True, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'in-network-nat', 'service_scaling': True, 'service_type': 'source-nat', 'service_virtualization_type': 'network-namespace', 'version': 1, 'vrouter_instance_type': None}, 'uuid': 'da51aa58-57d7-4e5a-a330-6a5968faa758'}, 'oper': 'CREATE', 'request-id': 'req-d4ae5fbc-0169-4e3e-b288-ade98ab37189', 'type': 'service_template', 'uuid': 'da51aa58-57d7-4e5a-a330-6a5968faa758'} 09/01/2025 01:56:18.821 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'haproxy-loadbalancer-template'] not found in DB: Unknown id: service_template default-domain:haproxy-loadbalancer-template 09/01/2025 01:56:18.821 7f7d9ddbd180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:haproxy-loadbalancer-template url = http://10.0.0.254:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.254 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "haproxy-loadbalancer-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "in-network-nat", "service_type": "loadbalancer", "image_name": null, "service_scaling": true, "interface_type": [{"service_interface_type": "right", "shared_ip": true, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": true, "static_route_enable": false}], "flavor": null, "ordered_interfaces": true, "service_virtualization_type": "network-namespace", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "27b8a85e67fe427d9ebc42e690325176", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "haproxy-loadbalancer-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:18.827 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:56:18.827 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:56:18.829 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:56:18.829 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:56:18.830 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:56:18.830 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:56:18.830 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:56:18.830 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:56:18.831 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:56:18.831 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:56:18.831 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:56:18.831 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:56:18.831 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:56:18.831 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:56:18.831 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:56:18.831 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:56:18.835 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:56:18.835 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:56:18.836 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:56:18.836 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:56:18.836 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:56:18.836 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:56:18.836 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:56:18.839 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:56:18.840 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:56:18.840 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:56:18.840 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:56:18.840 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:56:18.840 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:56:18.840 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:56:18.840 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:56:18.840 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:56:18.841 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:56:18.841 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:56:18.841 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:56:18.860 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'haproxy-loadbalancer-template'], 'obj_dict': {'display_name': 'haproxy-loadbalancer-template', 'fq_name': ['default-domain', 'haproxy-loadbalancer-template'], 'id_perms': {'created': '2025-09-01T01:56:18.854547', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:18.854547', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10847161831145534842, 'uuid_mslong': 303762325982299455}}, 'parent_type': 'domain', 'parent_uuid': 'ad8fac03-7fc3-4cff-8182-fd0fd71e19ae', 'perms2': {'global_access': 0, 'owner': '27b8a85e67fe427d9ebc42e690325176', 'owner_access': 7, 'share': [{'tenant': 'domain:ad8fac03-7fc3-4cff-8182-fd0fd71e19ae', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': None, 'image_name': None, 'instance_data': None, 'interface_type': [{'service_interface_type': 'right', 'shared_ip': True, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': True, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'in-network-nat', 'service_scaling': True, 'service_type': 'loadbalancer', 'service_virtualization_type': 'network-namespace', 'version': 1, 'vrouter_instance_type': None}, 'uuid': '04372e39-e052-453f-9688-dc36f69af57a'}, 'oper': 'CREATE', 'request-id': 'req-07e8e21c-07c2-4562-a00f-19b0459b8cfa', 'type': 'service_template', 'uuid': '04372e39-e052-453f-9688-dc36f69af57a'} 09/01/2025 01:56:18.915 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'docker-template'] not found in DB: Unknown id: service_template default-domain:docker-template 09/01/2025 01:56:18.920 7f7d9ddbd180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:docker-template url = http://10.0.0.254:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.254 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "docker-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "transparent", "service_type": "firewall", "image_name": "ubuntu", "service_scaling": false, "interface_type": [{"service_interface_type": "management", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "right", "shared_ip": false, "static_route_enable": false}], "flavor": null, "ordered_interfaces": true, "service_virtualization_type": "vrouter-instance", "availability_zone_enable": false, "vrouter_instance_type": "docker", "instance_data": "{\"command\":\"/bin/bash\"}"}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "27b8a85e67fe427d9ebc42e690325176", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "docker-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:18.941 7f7d9db39720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'] not found in DB: Unknown id: config_database_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1. 09/01/2025 01:56:18.949 7f7d9db39720 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = config_database_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1. url = http://10.0.0.254:8082/config-database-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_config_database_node.py remote_ip = 10.0.0.254 body = {"config-database-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1."], "parent_type": "global-system-config", "config_database_node_ip_address": "10.0.0.254", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3992-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:18.952 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'docker-template'], 'obj_dict': {'display_name': 'docker-template', 'fq_name': ['default-domain', 'docker-template'], 'id_perms': {'created': '2025-09-01T01:56:18.940161', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:18.940161', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13126405724707599779, 'uuid_mslong': 9645357154583659468}}, 'parent_type': 'domain', 'parent_uuid': 'ad8fac03-7fc3-4cff-8182-fd0fd71e19ae', 'perms2': {'global_access': 0, 'owner': '27b8a85e67fe427d9ebc42e690325176', 'owner_access': 7, 'share': [{'tenant': 'domain:ad8fac03-7fc3-4cff-8182-fd0fd71e19ae', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': None, 'image_name': 'ubuntu', 'instance_data': '{"command":"/bin/bash"}', 'interface_type': [{'service_interface_type': 'management', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'right', 'shared_ip': False, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'transparent', 'service_scaling': False, 'service_type': 'firewall', 'service_virtualization_type': 'vrouter-instance', 'version': 1, 'vrouter_instance_type': 'docker'}, 'uuid': '85db313a-92bb-43cc-b62a-5c75bed9b5a3'}, 'oper': 'CREATE', 'request-id': 'req-04376fa6-3d8b-4a6c-b84d-ebc97f2b42b0', 'type': 'service_template', 'uuid': '85db313a-92bb-43cc-b62a-5c75bed9b5a3'} 09/01/2025 01:56:18.962 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:18.975 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'], 'obj_dict': {'config_database_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3992-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'], 'id_perms': {'created': '2025-09-01T01:56:18.966770', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:18.966770', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12438079309716095845, 'uuid_mslong': 15090106233016174257}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd16ad3ab-09d6-46b1-ac9c-ef3b7fc38365'}, 'oper': 'CREATE', 'request-id': 'req-885a752f-3a1f-4cd8-b792-70db0881015f', 'type': 'config_database_node', 'uuid': 'd16ad3ab-09d6-46b1-ac9c-ef3b7fc38365'} 09/01/2025 01:56:18.976 7f7d9dbbc680 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:56:18.977 7f7d9e882d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:18.977 7f7d9e882d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. elements = { config_database_node_ip_address : "10.0.0.254"display_name : "cn-jenkins-deploy-platform-ansible-os-3992-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 15090106233016174257, "uuid_lslong": 12438079309716095845}, "enable": true, "created": "2025-09-01T01:56:18.966770", "last_modified": "2025-09-01T01:56:18.966770", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "d16ad3ab-09d6-46b1-ac9c-ef3b7fc38365" } deleted = False >> 09/01/2025 01:56:18.977 7f7d9e882d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:19.966 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:20.968 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:21.049 7f7d9db399a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_DNS_refs in resource network-ipam: 'virtual_DNS_refs' 09/01/2025 01:56:21.049 7f7d9db399a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource network-ipam: 'tag_refs' 09/01/2025 01:56:21.049 7f7d9db399a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_back_refs in resource network-ipam: 'virtual_network_back_refs' 09/01/2025 01:56:21.049 7f7d9db399a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_router_back_refs in resource network-ipam: 'virtual_router_back_refs' 09/01/2025 01:56:21.050 7f7d9db399a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource network-ipam: 'instance_ip_back_refs' 09/01/2025 01:56:21.354 7f7d9db390e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:21.355 7f7d9db390e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:21.356 7f7d9db390e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:21.356 7f7d9db390e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:21.356 7f7d9db390e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:21.356 7f7d9db390e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:21.356 7f7d9db390e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:21.357 7f7d9db390e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:21.357 7f7d9db390e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:21.357 7f7d9db390e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:21.357 7f7d9db390e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:21.357 7f7d9db390e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:21.358 7f7d9db390e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:56:21.358 7f7d9db390e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:21.636 7f7d9db399a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'] not found in DB: Unknown id: access_control_list default-domain:default-project:__no_rule__:ingress-access-control-list 09/01/2025 01:56:21.639 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:ingress-access-control-list url = http://10.0.0.254:8082/access-control-lists operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.254 body = {"access-control-list": {"uuid": null, "fq_name": ["default-domain", "default-project", "__no_rule__", "ingress-access-control-list"], "parent_type": "security-group", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "ingress-access-control-list"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:21.655 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:56:21.649544', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:21.649544', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13203257685329782482, 'uuid_mslong': 2528004923546619689}}, 'parent_type': 'security-group', 'parent_uuid': '9f2015c4-f792-480b-8162-094cf6d6ab73', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '23154715-826f-4f29-b73b-64e9340726d2'}, 'oper': 'CREATE', 'request-id': 'req-46b2d4bd-e723-451b-93e9-bc272888a376', 'type': 'access_control_list', 'uuid': '23154715-826f-4f29-b73b-64e9340726d2'} 09/01/2025 01:56:21.755 7f7d9db399a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'] not found in DB: Unknown id: access_control_list default-domain:default-project:__no_rule__:egress-access-control-list 09/01/2025 01:56:21.759 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:egress-access-control-list url = http://10.0.0.254:8082/access-control-lists operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.254 body = {"access-control-list": {"uuid": null, "fq_name": ["default-domain", "default-project", "__no_rule__", "egress-access-control-list"], "parent_type": "security-group", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "egress-access-control-list"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:21.777 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:56:21.770181', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:21.770181', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12720638828644713230, 'uuid_mslong': 13891977981821996092}}, 'parent_type': 'security-group', 'parent_uuid': '9f2015c4-f792-480b-8162-094cf6d6ab73', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c0ca385f-220c-4c3c-b088-c99e2968c30e'}, 'oper': 'CREATE', 'request-id': 'req-e1722877-1380-4bff-950d-5bd6bf5d810b', 'type': 'access_control_list', 'uuid': 'c0ca385f-220c-4c3c-b088-c99e2968c30e'} 09/01/2025 01:56:21.885 7f7d9db399a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000000'] not found in DB: Unknown id: route_target target:64512:8000000 09/01/2025 01:56:21.885 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000000 url = http://10.0.0.254:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.254 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000000"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "27b8a85e67fe427d9ebc42e690325176", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000000"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:21.899 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000000'], 'obj_dict': {'display_name': 'target:64512:8000000', 'fq_name': ['target:64512:8000000'], 'id_perms': {'created': '2025-09-01T01:56:21.892449', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:21.892449', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9906866012233534808, 'uuid_mslong': 1099819438931527018}}, 'perms2': {'global_access': 0, 'owner': '27b8a85e67fe427d9ebc42e690325176', 'owner_access': 7, 'share': []}, 'uuid': '0f4357f3-de95-456a-897c-422638920d58'}, 'oper': 'CREATE', 'request-id': 'req-8b8dd430-b9f8-4a92-af04-c5f4006ee126', 'type': 'route_target', 'uuid': '0f4357f3-de95-456a-897c-422638920d58'} 09/01/2025 01:56:21.910 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 3abc5d2d-5e7d-442a-b101-e08c43bc6ef8 object_type = routing_instance identifier_name = default-domain:default-project:dci-network:__default__ url = http://10.0.0.254:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.254 body = {"type": "routing-instance", "uuid": "3abc5d2d-5e7d-442a-b101-e08c43bc6ef8", "ref-type": "route-target", "ref-uuid": "0f4357f3-de95-456a-897c-422638920d58", "ref-fq-name": ["target:64512:8000000"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:21.920 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'oper': 'UPDATE', 'request-id': 'req-9fbf836e-8fdf-4d4e-89b2-4b7bec2c7e50', 'type': 'routing_instance', 'uuid': '3abc5d2d-5e7d-442a-b101-e08c43bc6ef8'} 09/01/2025 01:56:21.969 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:21.982 7f7d9db399a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000001'] not found in DB: Unknown id: route_target target:64512:8000001 09/01/2025 01:56:21.983 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000001 url = http://10.0.0.254:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.254 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000001"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "27b8a85e67fe427d9ebc42e690325176", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000001"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:22.000 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000001'], 'obj_dict': {'display_name': 'target:64512:8000001', 'fq_name': ['target:64512:8000001'], 'id_perms': {'created': '2025-09-01T01:56:21.990808', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:21.990808', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10275245882271155766, 'uuid_mslong': 982427147137075297}}, 'perms2': {'global_access': 0, 'owner': '27b8a85e67fe427d9ebc42e690325176', 'owner_access': 7, 'share': []}, 'uuid': '0da24849-02ce-4861-8e99-01b078914636'}, 'oper': 'CREATE', 'request-id': 'req-a900cab5-238e-4f77-8714-a9f66b5200ec', 'type': 'route_target', 'uuid': '0da24849-02ce-4861-8e99-01b078914636'} 09/01/2025 01:56:22.048 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_objects in resource global-vrouter-config: 'security_logging_objects' 09/01/2025 01:56:22.048 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-vrouter-config: 'tag_refs' 09/01/2025 01:56:22.048 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field application_policy_set_back_refs in resource global-vrouter-config: 'application_policy_set_back_refs' 09/01/2025 01:56:22.057 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 1ad7f85c-cbfe-4577-90e4-e28e82635f8c object_type = routing_instance identifier_name = default-domain:default-project:dci-network:dci-network url = http://10.0.0.254:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.254 body = {"type": "routing-instance", "uuid": "1ad7f85c-cbfe-4577-90e4-e28e82635f8c", "ref-type": "route-target", "ref-uuid": "0da24849-02ce-4861-8e99-01b078914636", "ref-fq-name": ["target:64512:8000001"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:22.066 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'oper': 'UPDATE', 'request-id': 'req-78a994d8-0626-475d-a558-d2ae3794ed7b', 'type': 'routing_instance', 'uuid': '1ad7f85c-cbfe-4577-90e4-e28e82635f8c'} 09/01/2025 01:56:22.125 7f7d9db399a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000002'] not found in DB: Unknown id: route_target target:64512:8000002 09/01/2025 01:56:22.125 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000002 url = http://10.0.0.254:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.254 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000002"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "27b8a85e67fe427d9ebc42e690325176", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000002"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:22.137 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000002'], 'obj_dict': {'display_name': 'target:64512:8000002', 'fq_name': ['target:64512:8000002'], 'id_perms': {'created': '2025-09-01T01:56:22.131768', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:22.131768', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11002289436871653977, 'uuid_mslong': 14679995526706056586}}, 'perms2': {'global_access': 0, 'owner': '27b8a85e67fe427d9ebc42e690325176', 'owner_access': 7, 'share': []}, 'uuid': 'cbb9d227-51a3-4d8a-98af-fbf340c47a59'}, 'oper': 'CREATE', 'request-id': 'req-07be16a9-933d-4b39-8ce1-25a61163cee6', 'type': 'route_target', 'uuid': 'cbb9d227-51a3-4d8a-98af-fbf340c47a59'} 09/01/2025 01:56:22.148 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 824fe8e8-2e04-42cf-81ac-5b76c7f8b538 object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.254:8082/global-vrouter-config/824fe8e8-2e04-42cf-81ac-5b76c7f8b538 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_linklocal.py remote_ip = 10.0.0.254 body = {"global-vrouter-config": {"uuid": "824fe8e8-2e04-42cf-81ac-5b76c7f8b538", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "linklocal_services": {"linklocal_service_entry": [{"linklocal_service_name": "metadata", "linklocal_service_ip": "169.254.169.254", "linklocal_service_port": 80, "ip_fabric_DNS_service_name": "", "ip_fabric_service_port": 8775, "ip_fabric_service_ip": ["10.0.0.254"]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:22.158 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-c39fa905-e0e8-476a-979a-054ff309ef44', 'type': 'global_vrouter_config', 'uuid': '824fe8e8-2e04-42cf-81ac-5b76c7f8b538'} 09/01/2025 01:56:22.191 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = d08c9dbf-20db-43cf-bfc7-0c46722f5453 object_type = routing_instance identifier_name = default-domain:default-project:default-virtual-network:default-virtual-network url = http://10.0.0.254:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.254 body = {"type": "routing-instance", "uuid": "d08c9dbf-20db-43cf-bfc7-0c46722f5453", "ref-type": "route-target", "ref-uuid": "cbb9d227-51a3-4d8a-98af-fbf340c47a59", "ref-fq-name": ["target:64512:8000002"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:22.205 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'oper': 'UPDATE', 'request-id': 'req-045732a8-49a7-48cb-9760-b91578d36dfe', 'type': 'routing_instance', 'uuid': 'd08c9dbf-20db-43cf-bfc7-0c46722f5453'} 09/01/2025 01:56:22.288 7f7d9db399a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000003'] not found in DB: Unknown id: route_target target:64512:8000003 09/01/2025 01:56:22.289 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000003 url = http://10.0.0.254:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.254 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000003"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "27b8a85e67fe427d9ebc42e690325176", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000003"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:22.309 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000003'], 'obj_dict': {'display_name': 'target:64512:8000003', 'fq_name': ['target:64512:8000003'], 'id_perms': {'created': '2025-09-01T01:56:22.302434', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:22.302434', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12058251247389599611, 'uuid_mslong': 1542603573894005145}}, 'perms2': {'global_access': 0, 'owner': '27b8a85e67fe427d9ebc42e690325176', 'owner_access': 7, 'share': []}, 'uuid': '15686dc7-0333-4199-a757-839f4511677b'}, 'oper': 'CREATE', 'request-id': 'req-55cc9b73-ec14-466a-b110-2e5f1b919a47', 'type': 'route_target', 'uuid': '15686dc7-0333-4199-a757-839f4511677b'} 09/01/2025 01:56:22.361 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = efa08bdb-dd5f-4027-83f4-45b15f3bb7b5 object_type = routing_instance identifier_name = default-domain:default-project:ip-fabric:ip-fabric url = http://10.0.0.254:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.254 body = {"type": "routing-instance", "uuid": "efa08bdb-dd5f-4027-83f4-45b15f3bb7b5", "ref-type": "route-target", "ref-uuid": "15686dc7-0333-4199-a757-839f4511677b", "ref-fq-name": ["target:64512:8000003"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:22.378 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-8edc46aa-663a-429f-9466-b9a90d021807', 'type': 'routing_instance', 'uuid': 'efa08bdb-dd5f-4027-83f4-45b15f3bb7b5'} 09/01/2025 01:56:22.510 7f7d9db399a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'] not found in DB: Unknown id: access_control_list default-domain:default-project:ip-fabric:ip-fabric 09/01/2025 01:56:22.519 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = access_control_list identifier_name = default-domain:default-project:ip-fabric:ip-fabric url = http://10.0.0.254:8082/access-control-lists operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.254 body = {"access-control-list": {"uuid": null, "fq_name": ["default-domain", "default-project", "ip-fabric", "ip-fabric"], "parent_type": "virtual-network", "access_control_list_entries": {"dynamic": false, "acl_rule": [{"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "deny", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000002", "direction": "<>"}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "ip-fabric"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:22.591 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'deny'}, 'direction': '<>', 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000002'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}], 'dynamic': False}, 'display_name': 'ip-fabric', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'id_perms': {'created': '2025-09-01T01:56:22.538553', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:22.538553', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9827072082500796160, 'uuid_mslong': 13982518562691301760}}, 'parent_type': 'virtual-network', 'parent_uuid': '5e021764-73b8-4f96-ab85-adf191959170', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c20be28c-da85-4180-8860-c5fe339e3f00'}, 'oper': 'CREATE', 'request-id': 'req-759ef23b-ed98-453e-9bb5-685ddec9bf73', 'type': 'access_control_list', 'uuid': 'c20be28c-da85-4180-8860-c5fe339e3f00'} 09/01/2025 01:56:22.649 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 23154715-826f-4f29-b73b-64e9340726d2 object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:ingress-access-control-list url = http://10.0.0.254:8082/access-control-list/23154715-826f-4f29-b73b-64e9340726d2 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.254 body = {"access-control-list": {"uuid": "23154715-826f-4f29-b73b-64e9340726d2", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "access_control_list_hash": 3520976145078303111}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:22.668 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-d788b31b-1bef-4f5c-b351-d3c8192c86ae', 'type': 'access_control_list', 'uuid': '23154715-826f-4f29-b73b-64e9340726d2'} 09/01/2025 01:56:22.716 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = c0ca385f-220c-4c3c-b088-c99e2968c30e object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:egress-access-control-list url = http://10.0.0.254:8082/access-control-list/c0ca385f-220c-4c3c-b088-c99e2968c30e operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.254 body = {"access-control-list": {"uuid": "c0ca385f-220c-4c3c-b088-c99e2968c30e", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "access_control_list_hash": 3520976145078303111}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:22.737 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-555ae619-cc7e-4a72-b825-06fd54ee8aae', 'type': 'access_control_list', 'uuid': 'c0ca385f-220c-4c3c-b088-c99e2968c30e'} 09/01/2025 01:56:22.846 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-conf-elements-inconsistency'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-conf-elements-inconsistency 09/01/2025 01:56:22.855 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-conf-elements-inconsistency url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-conf-elements-inconsistency"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements", "operation": "size==", "operand2": {"json_value": "1"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "ContrailConfig has a list of elements - it should be a dict. Controllers have inconsistency in their data.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-conf-elements-inconsistency"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:22.893 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements', 'operand2': {'json_value': '1'}, 'operation': 'size=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-elements-inconsistency', 'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'id_perms': {'created': '2025-09-01T01:56:22.874935', 'creator': None, 'description': 'ContrailConfig has a list of ' 'elements - it should be a dict. ' 'Controllers have inconsistency in ' 'their data.', 'enable': True, 'last_modified': '2025-09-01T01:56:22.874935', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13707213498574533941, 'uuid_mslong': 401551790739309311}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '05929939-e5f0-42ff-be39-ce0d92f97535', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-76d9f329-9cb9-4945-81da-f04173a8c033', 'type': 'alarm', 'uuid': '05929939-e5f0-42ff-be39-ce0d92f97535'} 09/01/2025 01:56:22.902 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-address-mismatch-compute'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-address-mismatch-compute 09/01/2025 01:56:22.910 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-address-mismatch-compute url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-address-mismatch-compute"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_ip_address", "operation": "!=", "operand2": {"uve_attribute": "VrouterAgent.control_ip"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Compute Node IP Address mismatch.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-address-mismatch-compute"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:22.940 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_ip_address', 'operand2': {'uve_attribute': 'VrouterAgent.control_ip'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-compute', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'id_perms': {'created': '2025-09-01T01:56:22.928755', 'creator': None, 'description': 'Compute Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:56:22.928755', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10598436790076336751, 'uuid_mslong': 4724792512980339102}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4191d5c1-ed9e-499e-9315-361b8eee6a6f', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-06fa47bc-0026-4b35-a7f8-174e556c2fcd', 'type': 'alarm', 'uuid': '4191d5c1-ed9e-499e-9315-361b8eee6a6f'} 09/01/2025 01:56:22.971 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:22.986 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-address-mismatch-control'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-address-mismatch-control 09/01/2025 01:56:22.991 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-address-mismatch-control url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-address-mismatch-control"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.bgp_router_parameters.address", "operation": "not in", "operand2": {"uve_attribute": "BgpRouterState.bgp_router_ip_list"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Control Node IP Address mismatch.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-address-mismatch-control"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:23.013 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.bgp_router_parameters.address', 'operand2': {'uve_attribute': 'BgpRouterState.bgp_router_ip_list'}, 'operation': 'not ' 'in'}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-control', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'id_perms': {'created': '2025-09-01T01:56:23.004550', 'creator': None, 'description': 'Control Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:56:23.004550', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10851083562822052220, 'uuid_mslong': 13252721359890825323}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b7eb1fdc-6386-406b-9696-cb024616a97c', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-e1a58d13-3fc6-415a-9b7e-f383e6d6d770', 'type': 'alarm', 'uuid': 'b7eb1fdc-6386-406b-9696-cb024616a97c'} 09/01/2025 01:56:23.059 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-bgp-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-bgp-connectivity 09/01/2025 01:56:23.065 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-bgp-connectivity url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-bgp-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "BgpRouterState.num_up_bgp_peer", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "BgpRouterState.num_up_bgp_peer", "operation": "!=", "operand2": {"uve_attribute": "BgpRouterState.num_bgp_peer"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "BGP peer mismatch. Not enough BGP peers are up.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-bgp-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:23.099 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_bgp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-bgp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'id_perms': {'created': '2025-09-01T01:56:23.084176', 'creator': None, 'description': 'BGP peer mismatch. Not enough BGP ' 'peers are up.', 'enable': True, 'last_modified': '2025-09-01T01:56:23.084176', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9736829477468174299, 'uuid_mslong': 8069936643805364712}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6ffe2c9f-05ed-41e8-8720-2ad2630af7db', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-54e2df4c-db1e-4aa3-a091-656c9b9c97e9', 'type': 'alarm', 'uuid': '6ffe2c9f-05ed-41e8-8720-2ad2630af7db'} 09/01/2025 01:56:23.147 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-conf-incorrect'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-conf-incorrect 09/01/2025 01:56:23.153 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-conf-incorrect url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-conf-incorrect"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig", "operation": "==", "operand2": {"json_value": "null"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-conf-incorrect"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:23.185 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-incorrect', 'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'id_perms': {'created': '2025-09-01T01:56:23.171905', 'creator': None, 'description': 'ContrailConfig missing or ' 'incorrect. Configuration pushed to ' 'Ifmap as ContrailConfig is ' 'missing/incorrect.', 'enable': True, 'last_modified': '2025-09-01T01:56:23.171905', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12769097842699676307, 'uuid_mslong': 5978628720978512785}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '52f85b25-45df-4b91-b134-f2d598522a93', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-498da01f-0c99-4312-a66d-aced82d86a1f', 'type': 'alarm', 'uuid': '52f85b25-45df-4b91-b134-f2d598522a93'} 09/01/2025 01:56:23.232 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-disk-usage-high'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-disk-usage-high 09/01/2025 01:56:23.240 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-disk-usage-high url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-disk-usage-high"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operation": "range", "operand2": {"json_value": "[70, 90]"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Disk usage crosses high threshold limit.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-disk-usage-high"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:23.271 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '[70, ' '90]'}, 'operation': 'range', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-disk-usage-high', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'id_perms': {'created': '2025-09-01T01:56:23.259561', 'creator': None, 'description': 'Disk usage crosses high threshold ' 'limit.', 'enable': True, 'last_modified': '2025-09-01T01:56:23.259561', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10860765991155787745, 'uuid_mslong': 269723242804036128}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '03be3fdc-3b00-4220-96b9-31204b7be7e1', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-4262ebc7-97f2-444e-8e55-ab4005e59e97', 'type': 'alarm', 'uuid': '03be3fdc-3b00-4220-96b9-31204b7be7e1'} 09/01/2025 01:56:23.315 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-disk-usage-critical'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-disk-usage-critical 09/01/2025 01:56:23.320 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-disk-usage-critical url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-disk-usage-critical"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operation": ">", "operand2": {"json_value": "90"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Disk usage crosses critical threshold limit.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-disk-usage-critical"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:23.343 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '90'}, 'operation': '>', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-disk-usage-critical', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'id_perms': {'created': '2025-09-01T01:56:23.334551', 'creator': None, 'description': 'Disk usage crosses critical ' 'threshold limit.', 'enable': True, 'last_modified': '2025-09-01T01:56:23.334551', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12789104832006102835, 'uuid_mslong': 5816089091841609693}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '50b6e636-cd3e-4fdd-b17c-0715caf49f33', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-18aa68f5-281b-4c9d-b8b8-968a1fad58a8', 'type': 'alarm', 'uuid': '50b6e636-cd3e-4fdd-b17c-0715caf49f33'} 09/01/2025 01:56:23.387 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-node-status'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-node-status 09/01/2025 01:56:23.392 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-node-status url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-node-status"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus", "operation": "==", "operand2": {"json_value": "null"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Node Failure. NodeStatus UVE not present.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-node-status"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:23.412 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-node-status', 'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'id_perms': {'created': '2025-09-01T01:56:23.404571', 'creator': None, 'description': 'Node Failure. NodeStatus UVE not ' 'present.', 'enable': True, 'last_modified': '2025-09-01T01:56:23.404571', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9323039891996661415, 'uuid_mslong': 11565774787221277774}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a081e2e4-107e-4c4e-8162-1762dd41d6a7', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-43e75ee2-0e81-4094-b851-383919996016', 'type': 'alarm', 'uuid': 'a081e2e4-107e-4c4e-8162-1762dd41d6a7'} 09/01/2025 01:56:23.455 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-partial-sysinfo'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-partial-sysinfo 09/01/2025 01:56:23.460 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-partial-sysinfo url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-partial-sysinfo"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.build_info", "operation": "==", "operand2": {"json_value": "null"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "System Info Incomplete.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-partial-sysinfo"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:23.477 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.build_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-partial-sysinfo', 'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'id_perms': {'created': '2025-09-01T01:56:23.469931', 'creator': None, 'description': 'System Info Incomplete.', 'enable': True, 'last_modified': '2025-09-01T01:56:23.469931', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9304822625772062689, 'uuid_mslong': 3067822192704113316}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '2a931807-be30-4aa4-8121-5ee10b2947e1', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-8b47ea0d-3da4-46c4-bfec-702a7665aa3f', 'type': 'alarm', 'uuid': '2a931807-be30-4aa4-8121-5ee10b2947e1'} 09/01/2025 01:56:23.522 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-process-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-process-connectivity 09/01/2025 01:56:23.526 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-process-connectivity url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-process-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.process_status", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "NodeStatus.process_status.state", "operation": "!=", "operand2": {"json_value": "\"Functional\""}, "variables": ["NodeStatus.process_status.module_id", "NodeStatus.process_status.instance_id"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Process(es) reporting as non-functional.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-process-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:23.541 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_status', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_status.state', 'operand2': {'json_value': '"Functional"'}, 'operation': '!=', 'variables': ['NodeStatus.process_status.module_id', 'NodeStatus.process_status.instance_id']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'id_perms': {'created': '2025-09-01T01:56:23.535766', 'creator': None, 'description': 'Process(es) reporting as ' 'non-functional.', 'enable': True, 'last_modified': '2025-09-01T01:56:23.535766', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11805350668708503676, 'uuid_mslong': 9027630143618696667}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '7d4895c9-0200-45db-a3d5-07e2c14a607c', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-8134d9f9-c370-4295-9172-309568ba76cd', 'type': 'alarm', 'uuid': '7d4895c9-0200-45db-a3d5-07e2c14a607c'} 09/01/2025 01:56:23.588 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-process-status'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-process-status 09/01/2025 01:56:23.592 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-process-status url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-process-status"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.process_info", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "NodeStatus.process_info.process_state", "operation": "!=", "operand2": {"json_value": "\"PROCESS_STATE_RUNNING\""}, "variables": ["NodeStatus.process_info.process_name"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Process Failure.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-process-status"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:23.615 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_info.process_state', 'operand2': {'json_value': '"PROCESS_STATE_RUNNING"'}, 'operation': '!=', 'variables': ['NodeStatus.process_info.process_name']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-status', 'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'id_perms': {'created': '2025-09-01T01:56:23.606325', 'creator': None, 'description': 'Process Failure.', 'enable': True, 'last_modified': '2025-09-01T01:56:23.606325', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9909032825796266958, 'uuid_mslong': 4775577704002765225}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4246429e-a45c-41a9-8983-f4dad1458bce', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-e71c5f18-61cc-4c06-ad1b-9d9699dfd807', 'type': 'alarm', 'uuid': '4246429e-a45c-41a9-8983-f4dad1458bce'} 09/01/2025 01:56:23.660 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-prouter-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-prouter-connectivity 09/01/2025 01:56:23.668 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-prouter-connectivity url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-prouter-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["prouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.connected_agent_list", "operation": "size!=", "operand2": {"json_value": "1"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Prouter connectivity to controlling tor agent does not exist we look for non-empty value for connected_agent_list", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-prouter-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:23.698 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.connected_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'id_perms': {'created': '2025-09-01T01:56:23.686927', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tor agent does not exist we look ' 'for non-empty value for ' 'connected_agent_list', 'enable': True, 'last_modified': '2025-09-01T01:56:23.686927', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11605371510303753011, 'uuid_mslong': 13942754282054962701}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c17e9d25-cc71-4e0d-a10e-8fe696180333', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-5fb42e9a-cf41-4e0d-a937-516a108a8f3b', 'type': 'alarm', 'uuid': 'c17e9d25-cc71-4e0d-a10e-8fe696180333'} 09/01/2025 01:56:23.744 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-prouter-tsn-connectivity 09/01/2025 01:56:23.754 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-prouter-tsn-connectivity url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-prouter-tsn-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["prouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.tsn_agent_list", "operation": "size!=", "operand2": {"json_value": "1"}}, {"operand1": "ProuterData.gateway_mode", "operation": "!=", "operand2": {"json_value": "\"SERVER\""}}]}, {"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.tsn_agent_list", "operation": "size!=", "operand2": {"json_value": "0"}}, {"operand1": "ProuterData.gateway_mode", "operation": "==", "operand2": {"json_value": "\"SERVER\""}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Prouter connectivity to controlling tsn agent does not exist we look for non-empty value for tsn_agent_list", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-prouter-tsn-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:23.831 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '!='}]}, {'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-tsn-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'id_perms': {'created': '2025-09-01T01:56:23.775191', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tsn agent does not exist we look ' 'for non-empty value for ' 'tsn_agent_list', 'enable': True, 'last_modified': '2025-09-01T01:56:23.775191', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11305459433293988792, 'uuid_mslong': 10975156150797551114}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '984f965e-cabb-460a-9ce5-0f74a813c3b8', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-b17726e5-5ef3-4c99-b321-94edb821d6de', 'type': 'alarm', 'uuid': '984f965e-cabb-460a-9ce5-0f74a813c3b8'} 09/01/2025 01:56:23.880 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-storage-cluster-state'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-storage-cluster-state 09/01/2025 01:56:23.890 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-storage-cluster-state url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-storage-cluster-state"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["storage-cluster"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "StorageCluster.info_stats.status", "operation": "!=", "operand2": {"json_value": "0"}, "variables": ["StorageCluster.info_stats.health_summary"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Storage Cluster warning/errors.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-storage-cluster-state"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:23.923 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'StorageCluster.info_stats.status', 'operand2': {'json_value': '0'}, 'operation': '!=', 'variables': ['StorageCluster.info_stats.health_summary']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-storage-cluster-state', 'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'id_perms': {'created': '2025-09-01T01:56:23.911037', 'creator': None, 'description': 'Storage Cluster warning/errors.', 'enable': True, 'last_modified': '2025-09-01T01:56:23.911037', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10764744346370981683, 'uuid_mslong': 12712140248492951266}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b06a9834-465e-46e2-9564-0df2ef563f33', 'uve_keys': {'uve_key': ['storage-cluster']}}, 'oper': 'CREATE', 'request-id': 'req-10419831-15d1-40bf-83ec-5472e84c971b', 'type': 'alarm', 'uuid': 'b06a9834-465e-46e2-9564-0df2ef563f33'} 09/01/2025 01:56:23.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:23.978 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-interface'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-interface 09/01/2025 01:56:23.985 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-interface url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-interface"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.down_interface_count", "operation": ">=", "operand2": {"json_value": "1"}, "variables": ["VrouterAgent.error_intf_list", "VrouterAgent.no_config_intf_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Vrouter interface(s) down.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-interface"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:24.019 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.down_interface_count', 'operand2': {'json_value': '1'}, 'operation': '>=', 'variables': ['VrouterAgent.error_intf_list', 'VrouterAgent.no_config_intf_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-interface', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'id_perms': {'created': '2025-09-01T01:56:24.005826', 'creator': None, 'description': 'Vrouter interface(s) down.', 'enable': True, 'last_modified': '2025-09-01T01:56:24.005826', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13267098328660119858, 'uuid_mslong': 12507144282433670629}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ad924d75-8d04-4de5-b81e-33a347f71132', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-fe1fd7c5-f3da-4cc1-bbf9-7e031b5ef6e8', 'type': 'alarm', 'uuid': 'ad924d75-8d04-4de5-b81e-33a347f71132'} 09/01/2025 01:56:24.063 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-xmpp-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-xmpp-connectivity 09/01/2025 01:56:24.069 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-xmpp-connectivity url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-xmpp-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "BgpRouterState.num_up_xmpp_peer", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "BgpRouterState.num_up_xmpp_peer", "operation": "!=", "operand2": {"uve_attribute": "BgpRouterState.num_xmpp_peer"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "XMPP peer mismatch.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-xmpp-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:24.093 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_xmpp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'id_perms': {'created': '2025-09-01T01:56:24.085409', 'creator': None, 'description': 'XMPP peer mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:56:24.085409', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10613316307313790156, 'uuid_mslong': 15048477777944463074}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd0d6eecf-0cbd-46e2-934a-12f3224c8ccc', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-86617697-b8e2-4808-9020-d97c77929c01', 'type': 'alarm', 'uuid': 'd0d6eecf-0cbd-46e2-934a-12f3224c8ccc'} 09/01/2025 01:56:24.141 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-xmpp-close-reason'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-xmpp-close-reason 09/01/2025 01:56:24.147 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-xmpp-close-reason url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-xmpp-close-reason"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["xmpp-peer"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "XmppPeerInfoData.close_reason", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "XmppPeerInfoData.state_info.state", "operation": "!=", "operand2": {"json_value": "\"Established\""}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "XMPP connection closed towards peer, alarm has close reason", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-xmpp-close-reason"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:24.179 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'XmppPeerInfoData.close_reason', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'XmppPeerInfoData.state_info.state', 'operand2': {'json_value': '"Established"'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-close-reason', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'id_perms': {'created': '2025-09-01T01:56:24.171114', 'creator': None, 'description': 'XMPP connection closed towards ' 'peer, ' 'alarm has close reason', 'enable': True, 'last_modified': '2025-09-01T01:56:24.171114', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10883433430889243905, 'uuid_mslong': 5037750801242341892}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '45e9afa9-9efd-4204-9709-b90ae0c3a501', 'uve_keys': {'uve_key': ['xmpp-peer']}}, 'oper': 'CREATE', 'request-id': 'req-1e1eb49c-bf50-4a0b-a98b-3143e3e7af31', 'type': 'alarm', 'uuid': '45e9afa9-9efd-4204-9709-b90ae0c3a501'} 09/01/2025 01:56:24.225 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-core-files'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-core-files 09/01/2025 01:56:24.231 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-core-files url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-core-files"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.all_core_file_list", "operand2": {"json_value": "null"}, "operation": "!="}, {"operand1": "NodeStatus.all_core_file_list", "operand2": {"json_value": "0"}, "operation": "size!="}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "A core file has been generated on the node.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-core-files"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:24.255 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-core-files', 'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'id_perms': {'created': '2025-09-01T01:56:24.243849', 'creator': None, 'description': 'A core file has been generated on ' 'the node.', 'enable': True, 'last_modified': '2025-09-01T01:56:24.243849', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13748926219152346334, 'uuid_mslong': 8461053901492079267}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '756bb3b1-dfde-4aa3-becd-ff8d27a14cde', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-3fbdd740-fd90-4e15-bfe8-ec9a33a0b985', 'type': 'alarm', 'uuid': '756bb3b1-dfde-4aa3-becd-ff8d27a14cde'} 09/01/2025 01:56:24.307 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-pending-cassandra-compaction-tasks 09/01/2025 01:56:24.315 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-pending-cassandra-compaction-tasks url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-pending-cassandra-compaction-tasks"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["database-nodeconfig-database-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks", "operand2": {"json_value": "300"}, "operation": ">="}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Pending compaction tasks in cassandra crossed the configured threshold.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-pending-cassandra-compaction-tasks"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:24.344 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks', 'operand2': {'json_value': '300'}, 'operation': '>='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-pending-cassandra-compaction-tasks', 'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'id_perms': {'created': '2025-09-01T01:56:24.333079', 'creator': None, 'description': 'Pending compaction tasks in ' 'cassandra crossed the configured ' 'threshold.', 'enable': True, 'last_modified': '2025-09-01T01:56:24.333079', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12891207850805120279, 'uuid_mslong': 16490720913298967380}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e4dacf4c-8f4a-4b54-b2e6-c53d60f74d17', 'uve_keys': {'uve_key': ['database-nodeconfig-database-node']}}, 'oper': 'CREATE', 'request-id': 'req-16c957d4-ef35-4d16-bc00-d578aebec520', 'type': 'alarm', 'uuid': 'e4dacf4c-8f4a-4b54-b2e6-c53d60f74d17'} 09/01/2025 01:56:24.389 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-package-version-mismatch'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-package-version-mismatch 09/01/2025 01:56:24.397 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-package-version-mismatch url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-package-version-mismatch"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.running_package_version", "operation": "!=", "operand2": {"uve_attribute": "NodeStatus.installed_package_version"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "There is a mismatch between installed and running package version.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-package-version-mismatch"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:24.429 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.running_package_version', 'operand2': {'uve_attribute': 'NodeStatus.installed_package_version'}, 'operation': '!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-package-version-mismatch', 'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'id_perms': {'created': '2025-09-01T01:56:24.417640', 'creator': None, 'description': 'There is a mismatch between ' 'installed and running package ' 'version.', 'enable': True, 'last_modified': '2025-09-01T01:56:24.417640', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10811203628475999079, 'uuid_mslong': 9751278193947004441}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '87537fda-5839-4e19-9609-1c6b7db9e367', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-4738224f-ef2d-48ba-b1e9-77c2b130e881', 'type': 'alarm', 'uuid': '87537fda-5839-4e19-9609-1c6b7db9e367'} 09/01/2025 01:56:24.479 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-limit-exceeded 09/01/2025 01:56:24.487 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-limit-exceeded url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-limit-exceeded"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.res_limit", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Agent resource usage exceeded configured watermark for resource.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-limit-exceeded"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:24.522 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'id_perms': {'created': '2025-09-01T01:56:24.510146', 'creator': None, 'description': 'Agent resource usage exceeded ' 'configured watermark for resource.', 'enable': True, 'last_modified': '2025-09-01T01:56:24.510146', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10402200013110493255, 'uuid_mslong': 5364210485056981659}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4a718103-ee3f-4a9b-905c-09cc7803c047', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-3319c6ba-ced4-4d6b-8a5d-c61eb5dbe705', 'type': 'alarm', 'uuid': '4a718103-ee3f-4a9b-905c-09cc7803c047'} 09/01/2025 01:56:24.579 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-table-limit-exceeded 09/01/2025 01:56:24.588 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-table-limit-exceeded url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-table-limit-exceeded"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.res_table_limit", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Agent resource usage exceeded table size for resource in vrouter.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-table-limit-exceeded"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:24.621 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_table_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-table-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'id_perms': {'created': '2025-09-01T01:56:24.608648', 'creator': None, 'description': 'Agent resource usage exceeded table ' 'size for resource in vrouter.', 'enable': True, 'last_modified': '2025-09-01T01:56:24.608648', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13099121594750311324, 'uuid_mslong': 7485663691657989736}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '67e26b77-8150-4e68-b5c9-6db034bd9b9c', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-31a0be84-5e88-4125-9e41-e5337ad2870d', 'type': 'alarm', 'uuid': '67e26b77-8150-4e68-b5c9-6db034bd9b9c'} 09/01/2025 01:56:24.669 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down 09/01/2025 01:56:24.677 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down url = http://10.0.0.254:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.254 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-l3mh-physical-interface-link-down"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.l3mh_phy_interface_down", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_l3mh_intf_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "On l3 multihomed compute one or more physical interfaces are Inactive.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-l3mh-physical-interface-link-down"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:24.707 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.l3mh_phy_interface_down', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_l3mh_intf_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-l3mh-physical-interface-link-down', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'id_perms': {'created': '2025-09-01T01:56:24.695200', 'creator': None, 'description': 'On l3 multihomed compute one or ' 'more physical interfaces are ' 'Inactive.', 'enable': True, 'last_modified': '2025-09-01T01:56:24.695200', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12204224033860072292, 'uuid_mslong': 12696640556796562436}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b0338751-14ad-4c04-a95e-1d191983b364', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-38436e15-27cc-4370-a094-93611edda716', 'type': 'alarm', 'uuid': 'b0338751-14ad-4c04-a95e-1d191983b364'} 09/01/2025 01:56:24.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:25.357 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:25.358 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:25.358 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:25.358 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:25.358 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:25.358 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:25.359 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:25.359 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:25.359 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:25.359 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:25.359 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:25.359 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:25.359 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:56:25.359 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:25.505 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 824fe8e8-2e04-42cf-81ac-5b76c7f8b538 object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.254:8082/global-vrouter-config/824fe8e8-2e04-42cf-81ac-5b76c7f8b538 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_encap.py remote_ip = 10.0.0.254 body = {"global-vrouter-config": {"uuid": "824fe8e8-2e04-42cf-81ac-5b76c7f8b538", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "encapsulation_priorities": {"encapsulation": ["MPLSoUDP", "MPLSoGRE", "VXLAN"]}, "vxlan_network_identifier_mode": "automatic"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:25.524 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-fd76ce42-d0e7-4391-8921-754ac861fdb2', 'type': 'global_vrouter_config', 'uuid': '824fe8e8-2e04-42cf-81ac-5b76c7f8b538'} 09/01/2025 01:56:25.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:26.274 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_objects in resource global-vrouter-config: 'security_logging_objects' 09/01/2025 01:56:26.275 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-vrouter-config: 'tag_refs' 09/01/2025 01:56:26.275 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field application_policy_set_back_refs in resource global-vrouter-config: 'application_policy_set_back_refs' 09/01/2025 01:56:26.382 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 824fe8e8-2e04-42cf-81ac-5b76c7f8b538 object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.254:8082/global-vrouter-config/824fe8e8-2e04-42cf-81ac-5b76c7f8b538 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_global_vrouter_config.py remote_ip = 10.0.0.254 body = {"global-vrouter-config": {"uuid": "824fe8e8-2e04-42cf-81ac-5b76c7f8b538", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "port_translation_pools": {"port_translation_pool": []}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:26.401 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-8668bce3-c2ac-4dad-858d-55d32c3b4029', 'type': 'global_vrouter_config', 'uuid': '824fe8e8-2e04-42cf-81ac-5b76c7f8b538'} 09/01/2025 01:56:26.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:27.173 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:56:27.174 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:56:27.174 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:56:27.174 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:56:27.174 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:56:27.174 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:56:27.174 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:56:27.174 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:56:27.174 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:56:27.174 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:56:27.175 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:56:27.175 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:56:27.175 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:56:27.175 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:56:27.175 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:56:27.175 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:56:27.175 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:56:27.175 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:56:27.175 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:56:27.175 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:56:27.176 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:56:27.176 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:56:27.176 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:56:27.176 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:56:27.176 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:56:27.176 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:56:27.176 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:56:27.176 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:56:27.176 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:56:27.176 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:56:27.176 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:56:27.177 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:56:27.177 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:56:27.177 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:56:27.177 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:56:27.232 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 38f19eae-05e2-496c-8f44-b9db734ce569 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.254:8082/global-system-config/38f19eae-05e2-496c-8f44-b9db734ce569 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.254 body = {"global-system-config": {"uuid": "38f19eae-05e2-496c-8f44-b9db734ce569", "autonomous_system": 64512, "ibgp_auto_mesh": true}} domain = default-domain project = admin user = admin >> 09/01/2025 01:56:27.268 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-7b99ca5f-3c9a-4d3c-b718-02b80f79f756', 'type': 'global_system_config', 'uuid': '38f19eae-05e2-496c-8f44-b9db734ce569'} 09/01/2025 01:56:27.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:28.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:29.359 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:29.361 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:29.362 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:29.362 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:29.362 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:29.363 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:29.363 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:29.363 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:29.364 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:29.364 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:29.364 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:29.365 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:29.365 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:56:29.365 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:29.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:30.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:31.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:32.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:33.362 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:33.363 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:33.363 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:33.363 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:33.363 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:33.364 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:33.364 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:33.364 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:33.364 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:33.364 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:33.364 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:33.364 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:33.365 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:56:33.365 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:33.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:34.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:35.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:36.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:37.364 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:37.366 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:37.366 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:37.366 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:37.367 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:37.367 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:37.367 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:37.368 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:37.368 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:37.369 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:37.369 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:37.369 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:37.369 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:56:37.369 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:37.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:39.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:40.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:41.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:41.367 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:41.369 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:41.370 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:41.370 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:41.371 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:41.371 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:41.371 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:41.373 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:41.373 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:41.374 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:41.374 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:41.374 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:41.374 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:56:41.375 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:42.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:43.022 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:44.023 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:45.371 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:45.373 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:45.374 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:45.374 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:45.375 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:45.375 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:45.375 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:45.376 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:45.377 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:45.377 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:45.378 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:45.378 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:45.378 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:56:45.378 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:45.952 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:46.953 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:47.955 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:48.956 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:49.374 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:49.377 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:49.378 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:49.378 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:49.378 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:49.378 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:49.379 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:49.379 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:49.379 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:49.380 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:49.380 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:49.380 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:49.380 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:56:49.380 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:49.957 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:50.959 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:51.961 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:52.962 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:53.378 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:53.379 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:53.379 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:53.380 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:53.380 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:53.380 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:53.380 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:53.380 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:53.380 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:53.380 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:53.380 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:53.380 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:53.380 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:56:53.380 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:53.964 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:54.966 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:55.967 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:56.968 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:57.381 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:57.383 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:57.383 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:57.383 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:57.384 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:57.384 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:57.384 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:57.385 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:57.385 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.385 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:57.385 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.386 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.386 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:56:57.386 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.970 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:58.972 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:59.974 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:00.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:01.383 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:01.386 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:01.386 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:01.386 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:01.387 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:01.387 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:01.387 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:01.388 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:01.389 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:01.389 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:01.389 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:01.389 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:01.390 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:01.390 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:01.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:02.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:03.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:04.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:05.386 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:05.389 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:05.389 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:05.389 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:05.390 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:05.390 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:05.390 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:05.391 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:05.391 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.392 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:05.392 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.392 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.392 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:05.393 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:06.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:07.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:08.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:09.389 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:09.392 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:09.392 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:09.392 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:09.393 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:09.393 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:09.393 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:09.394 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:09.394 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:09.395 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:09.395 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:09.395 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:09.395 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:09.396 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:09.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:10.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:11.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:12.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:13.392 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:13.393 7f7d9db89180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:13.393 7f7d9db89180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:13.393 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:13.393 7f7d9db89180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:13.393 7f7d9db89180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:13.394 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:13.394 7f7d9db89180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:13.394 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:13.394 7f7d9db89180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:13.394 7f7d9db89180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:13.394 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:13.394 7f7d9db89180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:13.394 7f7d9db89180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:13.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:15.952 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:16.954 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:17.394 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:17.396 7f7d9db89180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:17.396 7f7d9db89180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:17.396 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:17.397 7f7d9db89180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:17.397 7f7d9db89180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:17.397 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:17.398 7f7d9db89180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:17.398 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:17.399 7f7d9db89180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:17.399 7f7d9db89180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:17.399 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:17.399 7f7d9db89180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:17.399 7f7d9db89180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:17.956 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:18.957 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:19.959 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:20.961 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:21.397 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:21.399 7f7d9db89180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:21.400 7f7d9db89180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:21.400 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:21.400 7f7d9db89180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:21.400 7f7d9db89180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:21.400 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:21.401 7f7d9db89180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:21.401 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:21.402 7f7d9db89180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:21.402 7f7d9db89180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:21.402 7f7d9db89180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:21.402 7f7d9db89180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:21.402 7f7d9db89180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:21.963 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:22.965 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:23.967 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:24.968 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:25.400 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:25.402 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:25.403 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:25.403 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:25.403 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:25.404 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:25.404 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:25.405 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:25.405 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:25.406 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:25.406 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:25.406 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:25.406 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:25.406 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:25.970 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:26.973 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:27.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:28.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:29.403 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:29.406 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:29.406 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:29.407 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:29.407 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:29.407 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:29.407 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:29.408 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:29.409 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:29.409 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:29.409 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:29.409 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:29.410 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:29.410 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:29.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:30.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:31.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:32.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:33.407 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:33.409 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:33.410 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:33.410 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:33.410 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:33.410 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:33.411 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:33.411 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:33.412 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:33.412 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:33.412 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:33.412 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:33.413 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:33.413 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:33.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:34.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:35.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:36.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:37.410 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:37.412 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:37.412 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:37.413 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:37.413 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:37.413 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:37.413 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:37.414 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:37.414 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:37.415 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:37.415 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:37.415 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:37.415 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:37.415 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:37.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:38.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:39.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:41.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:41.413 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:41.414 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:41.414 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:41.414 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:41.414 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:41.415 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:41.415 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:41.415 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:41.415 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:41.415 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:41.415 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:41.415 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:41.415 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:41.416 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:42.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:42.502 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_routers in resource routing-instance: 'bgp_routers' 09/01/2025 01:57:42.503 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_refs in resource routing-instance: 'routing_instance_refs' 09/01/2025 01:57:42.503 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_target_refs in resource routing-instance: 'route_target_refs' 09/01/2025 01:57:42.503 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource routing-instance: 'tag_refs' 09/01/2025 01:57:42.503 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource routing-instance: 'virtual_machine_interface_back_refs' 09/01/2025 01:57:42.504 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_aggregate_back_refs in resource routing-instance: 'route_aggregate_back_refs' 09/01/2025 01:57:42.504 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_back_refs in resource routing-instance: 'routing_policy_back_refs' 09/01/2025 01:57:42.504 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_back_refs in resource routing-instance: 'routing_instance_back_refs' 09/01/2025 01:57:42.556 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3992-1'] not found in DB: Unknown id: bgp_router default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-ansible-os-3992-1 09/01/2025 01:57:42.561 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = bgp_router identifier_name = default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-ansible-os-3992-1 url = http://10.0.0.254:8082/bgp-routers operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.254 body = {"bgp-router": {"uuid": null, "fq_name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3992-1"], "parent_type": "routing-instance", "bgp_router_parameters": {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.17", "address": "10.20.0.17", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3992-1"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:42.592 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3992-1'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.17', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.17', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3992-1', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3992-1'], 'id_perms': {'created': '2025-09-01T01:57:42.581692', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:42.581692', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10345386454374284862, 'uuid_mslong': 3459903081964587736}}, 'parent_type': 'routing-instance', 'parent_uuid': '609baddc-83d5-45f8-96c3-3bbd47130ba3', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '30040b85-8dc7-4ad8-8f92-322b051dfe3e'}, 'oper': 'CREATE', 'request-id': 'req-2a5dfa96-b8dc-4981-82ac-28de68991641', 'type': 'bgp_router', 'uuid': '30040b85-8dc7-4ad8-8f92-322b051dfe3e'} 09/01/2025 01:57:42.592 7f7d9dbbc680 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/01/2025 01:57:42.592 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:42.593 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.17", "address": "10.20.0.17", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-ansible-os-3992-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3992-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3459903081964587736, "uuid_lslong": 10345386454374284862}, "enable": true, "created": "2025-09-01T01:57:42.581692", "last_modified": "2025-09-01T01:57:42.581692", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "609baddc-83d5-45f8-96c3-3bbd47130ba3"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "30040b85-8dc7-4ad8-8f92-322b051dfe3e" } deleted = False >> 09/01/2025 01:57:42.593 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:42.636 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zone_refs in resource bgp-router: 'control_node_zone_refs' 09/01/2025 01:57:42.636 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource bgp-router: 'sub_cluster_refs' 09/01/2025 01:57:42.636 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource bgp-router: 'bgp_router_refs' 09/01/2025 01:57:42.636 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource bgp-router: 'tag_refs' 09/01/2025 01:57:42.636 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_system_config_back_refs in resource bgp-router: 'global_system_config_back_refs' 09/01/2025 01:57:42.636 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource bgp-router: 'physical_router_back_refs' 09/01/2025 01:57:42.636 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource bgp-router: 'virtual_machine_interface_back_refs' 09/01/2025 01:57:42.637 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_as_a_service_back_refs in resource bgp-router: 'bgp_as_a_service_back_refs' 09/01/2025 01:57:42.637 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_back_refs in resource bgp-router: 'bgp_router_back_refs' 09/01/2025 01:57:42.738 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:42.738 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:42.738 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:42.739 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:42.739 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:42.739 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:42.739 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:42.739 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:42.739 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:42.739 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:42.739 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:42.739 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:42.739 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:42.740 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:42.740 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:42.740 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:42.740 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:42.740 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:42.740 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:42.740 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:42.740 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:42.740 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:42.740 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:42.741 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:42.741 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:42.741 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:42.741 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:42.741 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:42.741 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:42.741 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:42.741 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:42.741 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:42.741 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:57:42.742 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:42.742 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:42.792 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 38f19eae-05e2-496c-8f44-b9db734ce569 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.254:8082/ref-relax-for-delete operation = ref-relax-for-delete useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.254 body = {"uuid": "38f19eae-05e2-496c-8f44-b9db734ce569", "ref-uuid": "30040b85-8dc7-4ad8-8f92-322b051dfe3e"} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:42.846 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 38f19eae-05e2-496c-8f44-b9db734ce569 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.254:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.254 body = {"type": "global-system-config", "uuid": "38f19eae-05e2-496c-8f44-b9db734ce569", "ref-type": "bgp-router", "ref-uuid": "30040b85-8dc7-4ad8-8f92-322b051dfe3e", "ref-fq-name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3992-1"], "operation": "ADD", "attr": null} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:42.858 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-3ce5e647-c7bc-4b7d-bb0c-d2ff4f7f7bf8', 'type': 'global_system_config', 'uuid': '38f19eae-05e2-496c-8f44-b9db734ce569'} 09/01/2025 01:57:43.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:44.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:45.415 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:45.418 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:45.418 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:45.418 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:45.418 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:45.419 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:45.419 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:45.420 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:45.420 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:45.420 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:45.420 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:45.421 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:45.421 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:45.421 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:45.953 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:46.953 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:47.955 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:48.957 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:49.419 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:49.421 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:49.422 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:49.422 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:49.423 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:49.423 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:49.423 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:49.424 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:49.424 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.424 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:49.424 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.425 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.425 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:49.425 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.958 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:50.960 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:51.970 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:52.971 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:53.423 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:53.425 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:53.426 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:53.426 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:53.426 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:53.427 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:53.427 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:53.428 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:53.428 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:53.428 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:53.429 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:53.429 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:53.429 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:53.429 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:53.973 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:54.974 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:55.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:56.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:57.426 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:57.427 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:57.427 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:57.428 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:57.428 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:57.428 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:57.428 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:57.428 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:57.428 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:57.428 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:57.428 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:57.428 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:57.429 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:57.429 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:57.979 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:59.037 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:00.038 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:01.040 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:01.429 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:01.430 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:01.430 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:01.430 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:01.431 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:01.431 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:01.431 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:01.431 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:01.431 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:01.431 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:01.432 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:01.432 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:01.432 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:01.432 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:02.042 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:03.044 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:04.046 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:05.047 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:05.431 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:05.434 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:05.434 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:05.434 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:05.435 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:05.435 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:05.435 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:05.436 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:05.436 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:05.437 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:05.437 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:05.437 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:05.437 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:05.438 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:06.049 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:07.051 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:08.053 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:09.055 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:09.434 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:09.435 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:09.435 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:09.436 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:09.436 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:09.436 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:09.436 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:09.437 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:09.437 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:09.437 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:09.437 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:09.438 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:09.438 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:09.438 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:10.057 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:11.058 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:12.060 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:13.062 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:13.436 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:13.439 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:13.439 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:13.440 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:13.440 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:13.440 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:13.440 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:13.441 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:13.442 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:13.442 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:13.442 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:13.442 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:13.443 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:13.443 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:14.063 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:15.954 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:16.956 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:17.440 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:17.442 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:17.443 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:17.443 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:17.444 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:17.444 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:17.444 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:17.445 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:17.446 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:17.446 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:17.446 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:17.447 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:17.447 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:17.447 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:17.958 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:18.959 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:19.961 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:20.962 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:21.443 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:21.444 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:21.444 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:21.444 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:21.444 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:21.444 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:21.444 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:21.444 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:21.444 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:21.444 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:21.444 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:21.445 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:21.445 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:21.445 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:21.489 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:58:21.489 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:58:21.489 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:58:21.489 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:58:21.489 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:58:21.490 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:58:21.490 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:58:21.490 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:58:21.490 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:58:21.490 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:58:21.490 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:58:21.490 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:58:21.490 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:58:21.490 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:58:21.490 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:58:21.491 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:58:21.491 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:58:21.491 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:58:21.491 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:58:21.491 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:58:21.491 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:58:21.491 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:58:21.491 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:58:21.491 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:58:21.491 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:58:21.492 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:58:21.492 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:58:21.492 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:58:21.492 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:58:21.492 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:58:21.492 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:58:21.492 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:58:21.492 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:58:21.492 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:58:21.590 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'] not found in DB: Unknown id: database_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1. 09/01/2025 01:58:21.593 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = database_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1. url = http://10.0.0.254:8082/database-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_database_node.py remote_ip = 10.0.0.254 body = {"database-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1."], "parent_type": "global-system-config", "database_node_ip_address": "10.0.0.254", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3992-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:58:21.610 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'], 'obj_dict': {'database_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3992-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'], 'id_perms': {'created': '2025-09-01T01:58:21.603824', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:21.603824', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13522251961883716098, 'uuid_mslong': 6650891309445434345}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5c4cb668-733a-4be9-bba8-b083ef686a02'}, 'oper': 'CREATE', 'request-id': 'req-2eb6efc2-b65e-477d-875a-57bdd1f7b27a', 'type': 'database_node', 'uuid': '5c4cb668-733a-4be9-bba8-b083ef686a02'} 09/01/2025 01:58:21.611 7f7d9dbbc680 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:58:21.611 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:21.611 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. elements = { database_node_ip_address : "10.0.0.254"display_name : "cn-jenkins-deploy-platform-ansible-os-3992-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 6650891309445434345, "uuid_lslong": 13522251961883716098}, "enable": true, "created": "2025-09-01T01:58:21.603824", "last_modified": "2025-09-01T01:58:21.603824", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "5c4cb668-733a-4be9-bba8-b083ef686a02" } deleted = False >> 09/01/2025 01:58:21.611 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:21.964 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:22.966 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:23.967 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:24.968 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:25.445 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:25.446 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:25.448 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:25.448 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:25.448 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:25.448 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:25.448 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:25.449 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:25.449 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:25.450 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:25.450 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:25.450 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:25.451 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:25.451 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:25.970 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:26.972 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:27.974 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:28.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:29.447 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:29.449 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:29.449 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:29.449 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:29.449 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:29.449 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:29.450 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:29.450 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:29.450 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:29.450 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:29.451 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:29.451 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:29.451 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:29.451 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:29.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:30.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:31.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:32.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:33.450 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:33.452 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:33.452 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:33.452 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:33.453 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:33.453 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:33.453 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:33.454 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:33.454 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:33.454 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:33.454 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:33.455 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:33.455 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:33.455 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:33.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:34.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:35.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:36.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:37.453 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:37.455 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:37.455 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:37.455 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:37.455 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:37.455 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:37.455 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:37.456 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:37.456 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:37.456 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:37.456 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:37.456 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:37.456 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:37.457 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:37.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:38.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:39.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:41.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:41.456 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:41.457 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:41.457 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:41.457 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:41.457 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:41.457 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:41.457 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:41.458 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:41.458 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:41.458 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:41.458 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:41.458 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:41.458 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:41.458 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:42.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:43.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:44.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:45.458 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:45.460 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:45.460 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:45.460 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:45.461 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:45.461 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:45.461 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:45.462 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:45.462 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:45.462 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:45.463 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:45.463 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:45.463 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:45.463 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:45.955 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:46.957 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:47.959 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:48.961 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:49.460 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:49.463 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:49.463 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:49.463 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:49.464 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:49.464 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:49.464 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:49.465 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:49.465 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:49.466 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:49.466 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:49.466 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:49.466 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:49.467 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:49.962 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:50.963 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:51.964 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:52.966 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:53.464 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:53.465 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:53.466 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:53.466 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:53.466 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:53.466 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:53.466 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:53.466 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:53.467 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.467 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:53.467 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.467 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.467 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:53.467 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.968 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:54.969 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:55.971 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:56.973 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:57.466 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:57.468 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:57.469 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:57.469 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:57.470 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:57.470 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:57.470 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:57.471 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:57.471 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:57.471 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:57.471 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:57.472 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:57.472 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:57.472 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:57.974 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:58.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:59.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:00.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:01.470 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:01.472 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:01.473 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:01.473 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:01.473 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:01.474 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:01.474 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:01.475 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:01.475 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:01.475 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:01.475 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:01.476 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:01.476 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:01.476 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:01.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:02.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:03.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:04.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:05.472 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:05.475 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:05.475 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:05.475 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:05.475 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:05.475 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:05.475 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:05.476 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:05.476 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:05.476 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:05.476 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:05.476 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:05.476 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:05.476 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:05.806 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:59:05.807 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:59:05.807 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:59:05.807 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:59:05.808 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:59:05.808 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:59:05.808 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:59:05.808 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:59:05.809 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:59:05.809 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:59:05.809 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:59:05.809 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:59:05.810 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:59:05.810 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:59:05.810 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:59:05.810 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:59:05.811 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:59:05.811 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:59:05.811 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:59:05.811 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:59:05.812 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:59:05.812 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:59:05.812 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:59:05.812 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:59:05.813 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:59:05.813 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:59:05.813 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:59:05.813 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:59:05.814 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:59:05.814 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:59:05.814 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:59:05.814 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:59:05.815 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:59:05.815 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:59:05.918 7f7d9db89040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'] not found in DB: Unknown id: analytics_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1. 09/01/2025 01:59:05.927 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1. url = http://10.0.0.254:8082/analytics-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_analytics_node.py remote_ip = 10.0.0.254 body = {"analytics-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1."], "parent_type": "global-system-config", "analytics_node_ip_address": "10.0.0.254", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3992-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:59:05.957 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3992-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'], 'id_perms': {'created': '2025-09-01T01:59:05.943975', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:05.943975', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10801980332688434906, 'uuid_mslong': 5949089030653167583}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '528f68f4-2941-47df-95e8-57e198a666da'}, 'oper': 'CREATE', 'request-id': 'req-729ebb53-a7d3-4876-aa6e-45929115b4c8', 'type': 'analytics_node', 'uuid': '528f68f4-2941-47df-95e8-57e198a666da'} 09/01/2025 01:59:05.958 7f7d9dbbc680 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/01/2025 01:59:05.959 7f7d9d76b400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:05.960 7f7d9d76b400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. elements = { analytics_node_ip_address : "10.0.0.254"display_name : "cn-jenkins-deploy-platform-ansible-os-3992-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 5949089030653167583, "uuid_lslong": 10801980332688434906}, "enable": true, "created": "2025-09-01T01:59:05.943975", "last_modified": "2025-09-01T01:59:05.943975", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "528f68f4-2941-47df-95e8-57e198a666da" } deleted = False >> 09/01/2025 01:59:05.960 7f7d9d76b400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:05.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:06.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:07.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:08.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:09.475 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:09.476 7f7d9db89040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:09.476 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:09.477 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:09.477 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:09.477 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:09.477 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:09.477 7f7d9db89040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:09.477 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.478 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:09.478 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.478 7f7d9db89040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.478 7f7d9db89040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:09.478 7f7d9db89040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:10.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:11.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:13.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:13.477 7f7d9db39720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:13.478 7f7d9db39720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:13.479 7f7d9db39720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:13.479 7f7d9db39720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:13.479 7f7d9db39720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:13.479 7f7d9db39720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:13.479 7f7d9db39720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:13.479 7f7d9db39720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:13.479 7f7d9db39720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:13.479 7f7d9db39720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:13.480 7f7d9db39720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:13.480 7f7d9db39720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:13.480 7f7d9db39720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:13.480 7f7d9db39720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:15.956 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:16.958 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:17.480 7f7d9db39720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:17.483 7f7d9db39720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:17.483 7f7d9db39720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:17.483 7f7d9db39720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:17.484 7f7d9db39720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:17.484 7f7d9db39720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:17.484 7f7d9db39720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:17.485 7f7d9db39720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:17.485 7f7d9db39720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:17.486 7f7d9db39720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:17.486 7f7d9db39720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:17.486 7f7d9db39720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:17.487 7f7d9db39720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:17.487 7f7d9db39720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:17.960 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:18.962 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:19.964 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:20.965 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:21.483 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:21.485 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:21.485 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:21.485 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:21.485 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:21.485 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:21.485 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:21.486 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:21.486 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.486 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:21.486 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.486 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.486 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:21.486 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.967 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:22.969 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:23.972 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:24.974 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:25.486 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:25.500 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:25.500 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:25.501 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:25.501 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:25.501 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:25.502 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:25.502 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:25.503 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:25.503 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:25.503 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:25.503 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:25.504 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:25.504 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:25.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:26.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:27.979 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:28.443 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:59:28.443 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:59:28.443 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:59:28.443 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:59:28.443 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:59:28.443 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:59:28.443 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:59:28.443 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:59:28.443 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:59:28.443 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:59:28.444 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:59:28.445 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:59:28.445 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:59:28.445 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:59:28.445 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:59:28.445 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:59:28.445 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:59:28.445 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:59:28.445 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:59:28.445 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:59:28.546 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'] not found in DB: Unknown id: analytics_alarm_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1. 09/01/2025 01:59:28.552 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_alarm_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1. url = http://10.0.0.254:8082/analytics-alarm-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_analytics_alarm_node.py remote_ip = 10.0.0.254 body = {"analytics-alarm-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1."], "parent_type": "global-system-config", "analytics_alarm_node_ip_address": "10.0.0.254", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3992-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:59:28.577 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3992-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'], 'id_perms': {'created': '2025-09-01T01:59:28.568140', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:28.568140', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10362525483686141500, 'uuid_mslong': 5127378597392238562}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '47281bab-1681-47e2-8fcf-16063f0b9e3c'}, 'oper': 'CREATE', 'request-id': 'req-d758255e-9c4f-45d8-9754-4be8eda51a10', 'type': 'analytics_alarm_node', 'uuid': '47281bab-1681-47e2-8fcf-16063f0b9e3c'} 09/01/2025 01:59:28.577 7f7d9dbbc680 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/01/2025 01:59:28.578 7f7d9d76b540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:28.578 7f7d9d76b540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. elements = { analytics_alarm_node_ip_address : "10.0.0.254"display_name : "cn-jenkins-deploy-platform-ansible-os-3992-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 5127378597392238562, "uuid_lslong": 10362525483686141500}, "enable": true, "created": "2025-09-01T01:59:28.568140", "last_modified": "2025-09-01T01:59:28.568140", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "47281bab-1681-47e2-8fcf-16063f0b9e3c" } deleted = False >> 09/01/2025 01:59:28.578 7f7d9d76b540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:28.981 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:29.501 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:29.503 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:29.503 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:29.503 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:29.503 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:29.503 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:29.503 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:29.504 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:29.504 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:29.504 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:29.504 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:29.504 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:29.504 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:29.504 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:29.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:30.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:31.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:32.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:33.504 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:33.505 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:33.506 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:33.506 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:33.506 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:33.506 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:33.506 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:33.507 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:33.507 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:33.507 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:33.507 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:33.507 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:33.507 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:33.507 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:33.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:35.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:36.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:37.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:37.507 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:37.510 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:37.511 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:37.511 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:37.511 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:37.511 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:37.512 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:37.512 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:37.513 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:37.513 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:37.513 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:37.513 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:37.514 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:37.514 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:39.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:40.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:41.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:41.511 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:41.513 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:41.514 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:41.514 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:41.514 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:41.515 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:41.515 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:41.516 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:41.516 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:41.516 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:41.517 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:41.517 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:41.517 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:41.517 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:42.426 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:59:42.426 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:59:42.426 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:59:42.426 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:59:42.426 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:59:42.426 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:59:42.427 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:59:42.428 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:59:42.428 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:59:42.428 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:59:42.428 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:59:42.428 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:59:42.428 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:59:42.428 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:59:42.428 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:59:42.428 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:59:42.428 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:59:42.428 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:59:42.428 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:59:42.530 7f7d9db39e00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'] not found in DB: Unknown id: analytics_snmp_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1. 09/01/2025 01:59:42.537 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_snmp_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1. url = http://10.0.0.254:8082/analytics-snmp-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_analytics_snmp_node.py remote_ip = 10.0.0.254 body = {"analytics-snmp-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1."], "parent_type": "global-system-config", "analytics_snmp_node_ip_address": "10.0.0.254", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3992-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:59:42.559 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3992-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'], 'id_perms': {'created': '2025-09-01T01:59:42.551873', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:42.551873', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12682759219656008150, 'uuid_mslong': 2370119825507566310}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '20e45b6c-827d-42e6-b002-36506b6821d6'}, 'oper': 'CREATE', 'request-id': 'req-cc1fd9bc-3d21-4140-9d67-d1085a4f2f87', 'type': 'analytics_snmp_node', 'uuid': '20e45b6c-827d-42e6-b002-36506b6821d6'} 09/01/2025 01:59:42.559 7f7d9dbbc680 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 09/01/2025 01:59:42.560 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.560 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. elements = { analytics_snmp_node_ip_address : "10.0.0.254"display_name : "cn-jenkins-deploy-platform-ansible-os-3992-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 2370119825507566310, "uuid_lslong": 12682759219656008150}, "enable": true, "created": "2025-09-01T01:59:42.551873", "last_modified": "2025-09-01T01:59:42.551873", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "20e45b6c-827d-42e6-b002-36506b6821d6" } deleted = False >> 09/01/2025 01:59:42.560 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:43.015 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:44.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:45.515 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:45.516 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:45.517 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:45.517 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:45.517 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:45.517 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:45.517 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:45.518 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:45.518 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:45.518 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:45.518 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:45.519 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:45.519 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:45.519 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:45.958 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:46.971 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:47.972 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:48.974 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:49.517 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:49.518 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:49.518 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:49.518 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:49.519 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:49.519 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:49.519 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:49.519 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:49.519 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:49.519 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:49.519 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:49.519 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:49.519 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:49.520 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:49.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:50.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:51.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:52.981 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:53.519 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:53.520 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:53.520 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:53.520 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:53.520 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:53.520 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:53.521 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:53.521 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:53.521 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:53.521 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:53.521 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:53.521 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:53.521 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:53.521 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:53.983 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:54.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:55.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:56.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:57.521 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:57.523 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:57.524 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:57.524 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:57.524 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:57.525 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:57.525 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:57.526 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:57.526 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:57.526 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:57.526 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:57.527 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:57.527 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:57.527 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:57.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:58.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:59.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:00.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:01.525 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:01.527 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:01.527 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:01.527 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:01.527 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:01.527 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:01.528 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:01.528 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:01.528 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:01.529 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:01.529 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:01.529 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:01.529 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:01.529 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:01.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:02.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:04.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:05.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:05.528 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:05.529 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:05.529 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:05.529 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:05.530 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:05.530 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:05.530 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:05.530 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:05.530 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:05.530 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:05.531 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:05.531 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:05.531 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:05.531 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:07.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:08.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:09.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:09.530 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:09.531 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:09.532 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:09.532 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:09.532 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:09.532 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:09.532 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:09.533 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:09.533 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:09.533 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:09.533 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:09.533 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:09.534 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:09.534 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:11.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:12.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:13.015 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:13.533 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:13.534 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:13.534 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:13.534 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:13.534 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:13.534 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:13.535 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:13.535 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:13.535 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:13.535 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:13.535 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:13.535 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:13.535 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:13.536 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:15.958 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:16.960 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:17.535 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:17.536 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:17.537 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:17.537 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:17.537 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:17.537 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:17.537 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:17.538 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:17.538 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:17.538 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:17.538 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:17.538 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:17.538 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:17.538 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:17.962 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:18.964 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:19.965 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:20.967 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:21.546 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:21.548 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:21.548 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:21.548 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:21.549 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:21.549 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:21.549 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:21.550 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:21.550 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:21.551 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:21.551 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:21.551 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:21.551 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:21.552 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:21.969 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:22.970 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:23.972 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:24.973 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:25.549 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:25.550 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:25.550 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:25.550 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:25.550 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:25.551 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:25.551 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:25.551 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:25.551 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:25.551 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:25.551 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:25.551 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:25.551 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:25.552 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:25.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:26.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:27.979 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:28.981 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:29.551 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:29.552 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:29.553 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:29.553 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:29.553 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:29.553 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:29.553 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:29.554 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:29.554 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:29.554 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:29.554 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:29.554 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:29.554 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:29.554 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:31.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:32.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:33.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:33.553 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:33.554 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:33.555 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:33.555 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:33.555 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:33.555 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:33.555 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:33.555 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:33.555 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:33.556 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:33.556 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:33.556 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:33.556 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:33.556 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:35.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:36.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:37.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:37.555 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:37.558 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:37.558 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:37.559 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:37.559 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:37.559 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:37.560 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:37.560 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:37.561 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:37.561 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:37.561 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:37.562 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:37.562 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:37.562 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:39.019 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:40.021 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:41.022 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:41.559 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:41.561 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:41.561 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:41.561 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:41.561 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:41.561 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:41.561 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:41.562 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:41.562 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:41.562 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:41.562 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:41.562 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:41.562 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:41.562 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.025 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:43.027 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:44.029 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:45.561 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:45.563 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:45.563 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:45.563 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:45.563 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:45.563 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:45.563 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:45.564 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:45.564 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:45.564 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:45.564 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:45.564 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:45.564 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:45.564 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:45.959 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:46.960 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:47.962 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:48.962 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:49.565 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:49.568 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:49.569 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:49.569 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:49.570 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:49.570 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:49.570 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:49.571 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:49.572 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:49.572 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:49.572 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:49.573 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:49.573 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:49.573 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:49.963 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:50.965 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:51.967 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:52.969 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:53.570 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:53.573 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:53.573 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:53.573 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:53.574 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:53.574 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:53.574 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:53.575 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:53.575 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:53.576 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:53.576 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:53.576 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:53.577 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:53.577 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:53.972 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:54.974 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:55.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:56.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:57.573 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:57.576 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:57.576 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:57.576 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:57.577 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:57.578 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:57.578 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:57.580 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:57.580 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:57.581 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:57.582 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:57.582 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:57.583 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:57.584 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:57.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:58.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:59.981 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:00.983 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:01.576 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:01.578 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:01.578 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:01.578 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:01.578 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:01.578 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:01.578 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:01.579 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:01.579 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:01.579 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:01.579 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:01.579 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:01.579 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:01.579 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:01.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:02.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:03.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:04.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:05.579 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:05.581 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:05.582 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:05.582 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:05.582 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:05.582 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:05.582 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:05.583 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:05.583 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.583 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:05.583 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.584 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.584 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:05.584 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:06.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:07.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:08.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:09.582 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:09.585 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:09.585 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:09.585 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:09.586 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:09.586 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:09.586 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:09.587 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:09.587 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:09.588 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:09.588 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:09.588 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:09.588 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:09.589 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:09.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:11.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:12.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:13.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:13.585 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:13.588 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:13.588 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:13.588 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:13.589 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:13.589 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:13.589 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:13.590 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:13.590 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:13.590 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:13.591 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:13.591 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:13.591 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:13.591 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:15.960 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:16.962 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:17.589 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:17.591 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:17.591 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:17.592 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:17.592 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:17.592 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:17.593 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:17.593 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:17.594 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:17.594 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:17.594 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:17.595 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:17.595 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:17.595 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:17.964 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:18.966 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:19.968 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:20.970 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:21.593 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:21.595 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:21.595 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:21.595 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:21.596 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:21.596 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:21.596 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:21.597 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:21.597 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:21.598 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:21.598 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:21.598 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:21.599 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:21.599 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:21.972 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:22.973 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:23.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:24.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:25.596 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:25.598 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:25.598 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:25.598 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:25.599 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:25.599 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:25.599 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:25.600 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:25.600 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:25.600 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:25.600 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:25.601 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:25.601 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:25.601 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:25.979 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:26.981 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:27.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:28.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:29.598 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:29.600 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:29.601 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:29.601 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:29.602 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:29.602 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:29.602 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:29.603 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:29.603 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:29.604 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:29.604 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:29.604 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:29.604 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:29.605 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:29.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:30.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:31.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:32.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:33.602 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:33.604 7f7d9db39e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:33.605 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:33.605 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:33.606 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:33.606 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:33.606 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:33.607 7f7d9db39e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:33.607 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:33.608 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:33.608 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:33.608 7f7d9db39e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:33.608 7f7d9db39e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:33.609 7f7d9db39e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:33.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:34.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:35.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:36.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:37.606 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:37.607 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:37.608 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:37.608 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:37.608 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:37.608 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:37.608 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:37.609 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:37.609 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:37.609 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:37.609 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:37.609 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:37.609 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:37.610 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:39.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:40.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:41.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:41.609 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:41.610 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:41.610 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:41.610 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:41.610 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:41.610 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:41.610 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:41.610 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:41.611 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:41.611 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:41.611 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:41.611 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:41.611 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:41.611 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:43.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:44.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:45.611 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:45.613 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:45.614 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:45.614 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:45.614 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:45.614 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:45.615 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:45.615 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:45.616 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:45.616 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:45.616 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:45.617 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:45.617 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:45.617 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:45.961 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:46.962 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:47.964 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:48.966 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:49.614 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:49.617 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:49.617 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:49.618 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:49.618 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:49.618 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:49.618 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:49.619 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:49.620 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:49.620 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:49.620 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:49.620 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:49.621 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:49.621 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:49.967 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:50.970 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:51.972 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:52.973 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:53.618 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:53.620 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:53.620 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:53.621 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:53.621 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:53.621 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:53.622 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:53.622 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:53.623 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:53.623 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:53.623 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:53.623 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:53.624 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:53.624 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:53.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:54.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:55.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:56.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:57.621 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:57.622 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:57.622 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:57.622 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:57.623 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:57.623 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:57.623 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:57.623 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:57.623 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:57.623 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:57.623 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:57.624 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:57.624 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:57.624 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:57.983 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:58.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:59.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:00.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:01.624 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:01.626 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:01.627 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:01.627 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:01.627 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:01.627 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:01.628 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:01.628 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:01.629 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:01.629 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:01.629 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:01.629 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:01.630 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:01.630 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:01.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:02.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:03.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:04.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:05.628 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:05.629 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:05.629 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:05.629 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:05.629 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:05.630 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:05.630 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:05.630 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:05.630 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:05.630 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:05.630 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:05.630 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:05.631 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:05.631 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:05.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:07.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:08.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:09.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:09.630 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:09.632 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:09.632 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:09.632 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:09.632 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:09.632 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:09.632 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:09.633 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:09.633 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:09.633 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:09.633 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:09.633 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:09.633 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:09.634 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:11.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:12.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:13.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:13.633 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:13.635 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:13.636 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:13.636 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:13.636 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:13.636 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:13.637 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:13.637 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:13.638 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:13.638 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:13.638 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:13.639 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:13.639 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:13.639 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:15.962 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:16.964 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:17.636 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:17.638 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:17.639 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:17.639 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:17.639 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:17.639 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:17.639 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:17.640 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:17.640 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:17.640 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:17.640 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:17.640 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:17.640 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:17.640 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:17.965 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:18.967 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:19.969 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:20.971 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:21.640 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:21.642 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:21.642 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:21.642 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:21.642 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:21.643 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:21.643 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:21.643 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:21.643 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:21.644 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:21.644 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:21.644 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:21.644 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:21.644 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:21.973 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:22.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:23.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:24.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:25.643 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:25.645 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:25.645 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:25.646 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:25.646 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:25.646 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:25.646 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:25.647 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:25.647 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:25.648 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:25.648 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:25.648 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:25.648 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:25.649 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:25.979 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:26.981 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:27.983 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:28.135 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 02:02:28.136 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 02:02:28.136 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 02:02:28.136 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 02:02:28.136 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 02:02:28.137 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 02:02:28.137 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 02:02:28.137 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 02:02:28.137 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 02:02:28.138 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 02:02:28.138 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 02:02:28.138 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 02:02:28.138 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 02:02:28.138 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 02:02:28.139 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 02:02:28.139 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 02:02:28.139 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 02:02:28.139 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 02:02:28.140 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 02:02:28.140 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 02:02:28.140 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 02:02:28.140 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 02:02:28.140 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 02:02:28.141 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 02:02:28.141 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 02:02:28.141 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 02:02:28.141 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 02:02:28.142 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 02:02:28.142 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 02:02:28.142 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 02:02:28.142 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 02:02:28.143 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 02:02:28.143 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 02:02:28.143 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 02:02:28.254 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_routers in resource routing-instance: 'bgp_routers' 09/01/2025 02:02:28.254 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_refs in resource routing-instance: 'routing_instance_refs' 09/01/2025 02:02:28.254 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_target_refs in resource routing-instance: 'route_target_refs' 09/01/2025 02:02:28.255 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource routing-instance: 'tag_refs' 09/01/2025 02:02:28.255 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource routing-instance: 'virtual_machine_interface_back_refs' 09/01/2025 02:02:28.255 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_aggregate_back_refs in resource routing-instance: 'route_aggregate_back_refs' 09/01/2025 02:02:28.256 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_back_refs in resource routing-instance: 'routing_policy_back_refs' 09/01/2025 02:02:28.256 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_back_refs in resource routing-instance: 'routing_instance_back_refs' 09/01/2025 02:02:28.314 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 38f19eae-05e2-496c-8f44-b9db734ce569 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.254:8082/global-system-config/38f19eae-05e2-496c-8f44-b9db734ce569 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.254 body = {"global-system-config": {"uuid": "38f19eae-05e2-496c-8f44-b9db734ce569", "ip_fabric_subnets": {"subnet": [{"ip_prefix": "10.20.0.0", "ip_prefix_len": 24}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 02:02:28.334 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-72faeda3-d2b0-4e32-99fc-270f9df96566', 'type': 'global_system_config', 'uuid': '38f19eae-05e2-496c-8f44-b9db734ce569'} 09/01/2025 02:02:28.437 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1'] not found in DB: Unknown id: virtual_router default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1 09/01/2025 02:02:28.442 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = virtual_router identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1 url = http://10.0.0.254:8082/virtual-routers operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.254 body = {"virtual-router": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1"], "parent_type": "global-system-config", "virtual_router_dpdk_enabled": false, "virtual_router_ip_address": "10.20.0.17", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3992-1"}} domain = default-domain project = admin user = admin >> 09/01/2025 02:02:28.487 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1'], 'obj_dict': {'display_name': 'cn-jenkins-deploy-platform-ansible-os-3992-1', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1'], 'id_perms': {'created': '2025-09-01T02:02:28.460717', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:02:28.460717', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11539493739750933229, 'uuid_mslong': 13298779682352221738}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b88ec1a9-40c9-4e2a-a024-846addb2c2ed', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.17'}, 'oper': 'CREATE', 'request-id': 'req-82b84dd8-7f4e-413d-b483-2116dba14f65', 'type': 'virtual_router', 'uuid': 'b88ec1a9-40c9-4e2a-a024-846addb2c2ed'} 09/01/2025 02:02:28.488 7f7d9dbbc680 [contrail-api] [DEBUG]: Add uve in the [ObjectVRouter:ContrailConfigTrace] map 09/01/2025 02:02:28.490 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:28.490 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1 elements = { display_name : "cn-jenkins-deploy-platform-ansible-os-3992-1"fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 13298779682352221738, "uuid_lslong": 11539493739750933229}, "enable": true, "created": "2025-09-01T02:02:28.460717", "last_modified": "2025-09-01T02:02:28.460717", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "b88ec1a9-40c9-4e2a-a024-846addb2c2ed"virtual_router_dpdk_enabled : falsevirtual_router_ip_address : "10.20.0.17" } deleted = False >> 09/01/2025 02:02:28.490 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:28.591 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:02:28.592 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:02:28.592 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:02:28.592 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:02:28.592 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:02:28.592 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:02:28.593 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:02:28.593 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:02:28.755 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field access_control_lists in resource virtual-network: 'access_control_lists' 09/01/2025 02:02:28.756 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field floating_ip_pools in resource virtual-network: 'floating_ip_pools' 09/01/2025 02:02:28.756 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alias_ip_pools in resource virtual-network: 'alias_ip_pools' 09/01/2025 02:02:28.756 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instances in resource virtual-network: 'routing_instances' 09/01/2025 02:02:28.756 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bridge_domains in resource virtual-network: 'bridge_domains' 09/01/2025 02:02:28.756 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_object_refs in resource virtual-network: 'security_logging_object_refs' 09/01/2025 02:02:28.757 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_refs in resource virtual-network: 'qos_config_refs' 09/01/2025 02:02:28.757 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-network: 'network_ipam_refs' 09/01/2025 02:02:28.757 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_policy_refs in resource virtual-network: 'network_policy_refs' 09/01/2025 02:02:28.757 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_refs in resource virtual-network: 'virtual_network_refs' 09/01/2025 02:02:28.757 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_health_check_refs in resource virtual-network: 'service_health_check_refs' 09/01/2025 02:02:28.757 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_table_refs in resource virtual-network: 'route_table_refs' 09/01/2025 02:02:28.757 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field multicast_policy_refs in resource virtual-network: 'multicast_policy_refs' 09/01/2025 02:02:28.758 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgpvpn_refs in resource virtual-network: 'bgpvpn_refs' 09/01/2025 02:02:28.758 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_map_refs in resource virtual-network: 'intent_map_refs' 09/01/2025 02:02:28.758 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_refs in resource virtual-network: 'routing_policy_refs' 09/01/2025 02:02:28.758 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-network: 'tag_refs' 09/01/2025 02:02:28.758 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_back_refs in resource virtual-network: 'virtual_network_back_refs' 09/01/2025 02:02:28.758 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource virtual-network: 'virtual_machine_interface_back_refs' 09/01/2025 02:02:28.758 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-network: 'instance_ip_back_refs' 09/01/2025 02:02:28.759 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-network: 'physical_router_back_refs' 09/01/2025 02:02:28.759 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field port_tuple_back_refs in resource virtual-network: 'port_tuple_back_refs' 09/01/2025 02:02:28.759 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field logical_router_back_refs in resource virtual-network: 'logical_router_back_refs' 09/01/2025 02:02:28.759 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_node_back_refs in resource virtual-network: 'flow_node_back_refs' 09/01/2025 02:02:28.759 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field firewall_rule_back_refs in resource virtual-network: 'firewall_rule_back_refs' 09/01/2025 02:02:28.759 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnect_back_refs in resource virtual-network: 'data_center_interconnect_back_refs' 09/01/2025 02:02:28.759 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabric_back_refs in resource virtual-network: 'fabric_back_refs' 09/01/2025 02:02:28.759 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field host_based_service_back_refs in resource virtual-network: 'host_based_service_back_refs' 09/01/2025 02:02:28.816 7f7d9db39ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1', 'vhost0'] not found in DB: Unknown id: virtual_machine_interface default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1:vhost0 09/01/2025 02:02:28.825 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = virtual_machine_interface identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1:vhost0 url = http://10.0.0.254:8082/virtual-machine-interfaces operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.254 body = {"virtual-machine-interface": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1", "vhost0"], "parent_type": "virtual-router", "virtual_machine_interface_disable_policy": true, "virtual_network_refs": [{"to": ["default-domain", "default-project", "ip-fabric"], "uuid": "5e021764-73b8-4f96-ab85-adf191959170"}], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "vhost0"}} domain = default-domain project = admin user = admin >> 09/01/2025 02:02:28.863 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:02:28.846248', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:02:28.846248', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9876995726292915272, 'uuid_mslong': 118707417928581946}}, 'parent_type': 'virtual-router', 'parent_uuid': 'b88ec1a9-40c9-4e2a-a024-846addb2c2ed', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '01a5bbc4-85d6-433a-8912-234851bd9448', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:01:a5:bb:c4:85']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '5e021764-73b8-4f96-ab85-adf191959170'}]}, 'oper': 'CREATE', 'request-id': 'req-4a51e9f4-714f-49fc-a54a-e7ab046530de', 'type': 'virtual_machine_interface', 'uuid': '01a5bbc4-85d6-433a-8912-234851bd9448'} 09/01/2025 02:02:28.863 7f7d9dbbc680 [contrail-api] [DEBUG]: Add uve in the [ObjectVMITable:ContrailConfigTrace] map 09/01/2025 02:02:28.864 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:28.865 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 118707417928581946, "uuid_lslong": 9876995726292915272}, "enable": true, "created": "2025-09-01T02:02:28.846248", "last_modified": "2025-09-01T02:02:28.846248", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "b88ec1a9-40c9-4e2a-a024-846addb2c2ed"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "01a5bbc4-85d6-433a-8912-234851bd9448"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:01:a5:bb:c4:85"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "uuid": "5e021764-73b8-4f96-ab85-adf191959170"}] } deleted = False >> 09/01/2025 02:02:28.865 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:28.875 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 01a5bbc4-85d6-433a-8912-234851bd9448 object_type = virtual_machine_interface identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1:vhost0 url = http://127.0.0.1/ref-update operation = ref-update domain = default-domain >> 09/01/2025 02:02:28.890 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-4a51e9f4-714f-49fc-a54a-e7ab046530de', 'type': 'virtual_machine_interface', 'uuid': '01a5bbc4-85d6-433a-8912-234851bd9448'} 09/01/2025 02:02:28.894 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:28.894 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3992-1:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3992-1", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 118707417928581946, "uuid_lslong": 9876995726292915272}, "enable": true, "created": "2025-09-01T02:02:28.846248", "last_modified": "2025-09-01T02:02:28.880025", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "b88ec1a9-40c9-4e2a-a024-846addb2c2ed"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}routing_instance_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "ip-fabric"], "attr": {"direction": "both", "vlan_tag": null, "src_mac": null, "dst_mac": null, "mpls_label": null, "service_chain_address": null, "ipv6_service_chain_address": null, "protocol": null}, "uuid": "efa08bdb-dd5f-4027-83f4-45b15f3bb7b5"}]uuid : "01a5bbc4-85d6-433a-8912-234851bd9448"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:01:a5:bb:c4:85"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "attr": null, "uuid": "5e021764-73b8-4f96-ab85-adf191959170"}] } deleted = False >> 09/01/2025 02:02:28.894 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:28.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:29.452 7f7d9db399a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = c20be28c-da85-4180-8860-c5fe339e3f00 object_type = access_control_list identifier_name = default-domain:default-project:ip-fabric:ip-fabric url = http://10.0.0.254:8082/access-control-list/c20be28c-da85-4180-8860-c5fe339e3f00 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3992-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.254 body = {"access-control-list": {"uuid": "c20be28c-da85-4180-8860-c5fe339e3f00", "access_control_list_entries": {"dynamic": false, "acl_rule": [{"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "deny", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000002", "direction": "<>"}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}]}, "access_control_list_hash": 9149612269987441733}} domain = default-domain project = admin user = admin >> 09/01/2025 02:02:29.476 7f7d9dbbc680 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-258ee4b7-b364-49f8-ace1-a85ef0cecfe7', 'type': 'access_control_list', 'uuid': 'c20be28c-da85-4180-8860-c5fe339e3f00'} 09/01/2025 02:02:29.646 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:29.648 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:29.649 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:29.649 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:29.650 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:29.650 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:29.650 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:29.651 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:29.651 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:29.651 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:29.652 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:29.652 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:29.652 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:29.652 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:29.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:30.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:31.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:32.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:33.650 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:33.651 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:33.652 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:33.652 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:33.652 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:33.652 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:33.653 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:33.653 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:33.653 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:33.654 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:33.654 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:33.654 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:33.654 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:33.654 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:33.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:34.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:35.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:36.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:37.652 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:37.654 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:37.654 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:37.654 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:37.654 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:37.654 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:37.654 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:37.655 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:37.655 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:37.655 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:37.655 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:37.655 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:37.655 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:37.655 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:39.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:40.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:40.746 7f7d9db39ea0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:02:40.756 7f7d9db39ea0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:02:41.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:41.654 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:41.655 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:41.655 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:41.655 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:41.656 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:41.656 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:41.656 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:41.656 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:41.656 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:41.656 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:41.656 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:41.656 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:41.657 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:41.657 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:43.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:44.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:45.657 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:45.659 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:45.659 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:45.660 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:45.660 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:45.660 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:45.660 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:45.661 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:45.662 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:45.662 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:45.662 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:45.662 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:45.663 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:45.663 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:45.963 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:46.965 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:47.967 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:48.968 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:49.660 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:49.664 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:49.664 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:49.664 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:49.665 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:49.665 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:49.665 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:49.666 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:49.667 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:49.667 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:49.667 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:49.667 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:49.668 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:49.668 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:49.971 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:50.973 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:51.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:52.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:53.628 7f7d9db39ea0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:02:53.636 7f7d9db39ea0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:02:53.664 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:53.666 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:53.666 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:53.667 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:53.667 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:53.667 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:53.668 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:53.668 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:53.669 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:53.669 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:53.669 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:53.669 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:53.670 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:53.670 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:53.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:54.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:55.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:56.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:57.667 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:57.668 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:57.669 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:57.669 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:57.669 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:57.670 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:57.670 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:57.671 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:57.671 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:57.671 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:57.672 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:57.672 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:57.672 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:57.672 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:57.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:58.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:59.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:00.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:01.670 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:01.672 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:01.673 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:01.673 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:01.673 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:01.674 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:01.674 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:01.675 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:01.675 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:01.675 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:01.676 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:01.676 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:01.676 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:01.677 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:01.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:02.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:04.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:05.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:05.673 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:05.675 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:05.676 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:05.676 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:05.676 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:05.677 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:05.677 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:05.678 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:05.678 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:05.678 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:05.678 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:05.679 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:05.679 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:05.679 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:06.655 7f7d9db39ea0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:03:06.663 7f7d9db39ea0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:03:07.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:08.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:09.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:09.676 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:09.678 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:09.679 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:09.679 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:09.680 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:09.680 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:09.680 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:09.681 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:09.681 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:09.682 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:09.682 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:09.682 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:09.682 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:09.683 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:11.015 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:12.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:13.019 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:13.680 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:13.682 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:13.683 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:13.683 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:13.683 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:13.684 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:13.684 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:13.685 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:13.685 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:13.685 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:13.686 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:13.686 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:13.686 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:13.686 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.021 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:15.964 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:16.966 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:17.683 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:17.686 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:17.686 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:17.687 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:17.687 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:17.687 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:17.687 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:17.688 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:17.689 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:17.689 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:17.689 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:17.690 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:17.690 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:17.690 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:17.967 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:18.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:19.653 7f7d9db39ea0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:03:19.663 7f7d9db39ea0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:03:19.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:20.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:21.687 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:21.688 7f7d9db39ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:21.688 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:21.688 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:21.688 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:21.688 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:21.689 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:21.689 7f7d9db39ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:21.689 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:21.689 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:21.689 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:21.689 7f7d9db39ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:21.689 7f7d9db39ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:21.690 7f7d9db39ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:21.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:22.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:23.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:24.486 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:03:24.487 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:03:24.487 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:03:24.488 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:03:24.488 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:03:24.488 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:03:24.489 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:03:24.489 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:03:24.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:25.688 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:25.691 7f7d9d76b7c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:25.693 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:25.693 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:25.694 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:25.694 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:25.694 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:25.695 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:25.695 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:25.695 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:25.696 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:25.696 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:25.696 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:25.696 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:25.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:26.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:27.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:28.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:29.693 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:29.694 7f7d9d76b7c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:29.694 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:29.694 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:29.694 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:29.694 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:29.695 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:29.695 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:29.695 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:29.695 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:29.695 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:29.695 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:29.695 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:29.695 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:29.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:30.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:31.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:32.507 7f7d9d76b7c0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:03:32.515 7f7d9d76b7c0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:03:33.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:33.695 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:33.697 7f7d9d76b7c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:33.698 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:33.698 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:33.698 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:33.699 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:33.699 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:33.700 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:33.700 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:33.700 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:33.700 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:33.701 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:33.701 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:33.701 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:35.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:36.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:37.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:37.698 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:37.700 7f7d9d76b7c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:37.700 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:37.700 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:37.701 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:37.701 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:37.701 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:37.702 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:37.702 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:37.703 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:37.703 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:37.703 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:37.703 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:37.704 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:39.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:40.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:41.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:41.701 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:41.703 7f7d9d76b7c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:41.703 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:41.704 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:41.704 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:41.704 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:41.705 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:41.705 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:41.706 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:41.706 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:41.706 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:41.707 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:41.707 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:41.707 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.018 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:43.020 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:44.022 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:45.704 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:45.707 7f7d9d76b7c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:45.708 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:45.708 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:45.708 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:45.708 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:45.709 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:45.710 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:45.710 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:45.710 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:45.710 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:45.711 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:45.711 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:45.711 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:45.847 7f7d9db39f40 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:03:45.856 7f7d9db39f40 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:03:45.965 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:46.966 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:47.967 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:48.969 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:49.709 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:49.711 7f7d9db39f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:49.712 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:49.712 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:49.713 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:49.713 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:49.713 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:49.714 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:49.714 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:49.715 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:49.715 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:49.715 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:49.715 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:49.716 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:49.970 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:50.972 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:51.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:52.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:53.713 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:53.714 7f7d9db39f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:53.715 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:53.715 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:53.715 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:53.715 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:53.716 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:53.716 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:53.716 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:53.717 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:53.717 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:53.717 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:53.717 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:53.717 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:53.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:54.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:55.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:56.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:57.716 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:57.719 7f7d9db39f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:57.719 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:57.719 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:57.720 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:57.720 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:57.720 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:57.721 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:57.722 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:57.722 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:57.723 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:57.723 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:57.723 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:57.724 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:57.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:58.614 7f7d9db39f40 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:03:58.620 7f7d9db39f40 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:03:58.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:59.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:00.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:01.720 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:01.721 7f7d9db39f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:01.722 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:01.722 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:01.722 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:01.722 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:01.722 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:01.722 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:01.722 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:01.723 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:01.723 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:01.723 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:01.723 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:01.723 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:01.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:02.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:03.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:05.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:05.723 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:05.726 7f7d9db39f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:05.726 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:05.726 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:05.727 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:05.727 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:05.727 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:05.728 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:05.728 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:05.729 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:05.729 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:05.729 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:05.730 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:05.730 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:07.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:08.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:09.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:09.726 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:09.802 7f7d9db39f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:09.802 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:09.802 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:09.802 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:09.803 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:09.803 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:09.803 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:09.804 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:09.804 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:09.804 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:09.804 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:09.804 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:09.805 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:11.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:11.627 7f7d9db39f40 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:04:11.637 7f7d9db39f40 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:04:12.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:13.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:13.802 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:13.805 7f7d9db39f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:13.805 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:13.806 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:13.806 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:13.807 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:13.807 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:13.808 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:13.808 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:13.808 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:13.808 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:13.809 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:13.809 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:13.809 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:15.966 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:16.968 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:17.806 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:17.808 7f7d9db39f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:17.809 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:17.809 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:17.809 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:17.810 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:17.810 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:17.811 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:17.811 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:17.812 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:17.812 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:17.812 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:17.813 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:17.813 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:17.970 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:18.973 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:19.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:20.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:21.810 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:21.812 7f7d9db39f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:21.813 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:21.813 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:21.813 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:21.814 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:21.814 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:21.815 7f7d9db39f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:21.815 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:21.816 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:21.816 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:21.816 7f7d9db39f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:21.816 7f7d9db39f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:21.817 7f7d9db39f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:21.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:22.981 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:23.983 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:24.427 7f7d9db39f40 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:04:24.435 7f7d9db39f40 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:04:24.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:25.459 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:04:25.460 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:04:25.460 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:04:25.460 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:04:25.460 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:04:25.460 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:04:25.460 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:04:25.460 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:04:25.813 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:25.815 7f7d9d76b7c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:25.816 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:25.816 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:25.816 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:25.817 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:25.817 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:25.818 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:25.818 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:25.818 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:25.818 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:25.819 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:25.819 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:25.819 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:25.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:26.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:27.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:28.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:29.816 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:29.817 7f7d9d76b7c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:29.817 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:29.817 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:29.817 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:29.818 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:29.818 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:29.818 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:29.818 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:29.818 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:29.818 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:29.818 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:29.818 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:29.818 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:29.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:30.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:31.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:33.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:33.818 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:33.820 7f7d9d76b7c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:33.821 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:33.821 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:33.821 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:33.821 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:33.822 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:33.823 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:33.823 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:33.823 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:33.824 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:33.824 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:33.824 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:33.825 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:35.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:36.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:37.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:37.273 7f7d9d76b7c0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:04:37.279 7f7d9d76b7c0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:04:37.821 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:37.823 7f7d9d76b7c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:37.823 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:37.823 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:37.824 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:37.824 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:37.824 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:37.825 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:37.825 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:37.825 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:37.825 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:37.825 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:37.826 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:37.826 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:39.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:40.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:41.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:41.824 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:41.830 7f7d9d76b7c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:41.830 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:41.831 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:41.831 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:41.832 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:41.832 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:41.833 7f7d9d76b7c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:41.833 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:41.834 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:41.834 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:41.834 7f7d9d76b7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:41.835 7f7d9d76b7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:41.835 7f7d9d76b7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:43.018 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:44.020 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:45.831 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:45.834 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:45.834 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:45.834 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:45.835 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:45.835 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:45.835 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:45.836 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:45.836 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:45.836 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:45.836 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:45.837 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:45.837 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:45.837 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:45.967 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:46.968 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:47.972 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:48.974 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:49.835 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:49.837 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:49.837 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:49.838 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:49.838 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:49.838 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:49.838 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:49.840 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:49.840 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:49.840 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:49.841 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:49.841 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:49.841 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:49.841 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:49.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:50.512 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:04:50.515 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:04:50.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:51.979 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:52.981 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:53.838 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:53.840 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:53.840 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:53.840 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:53.840 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:53.841 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:53.841 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:53.841 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:53.842 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:53.842 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:53.842 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:53.842 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:53.842 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:53.842 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:53.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:54.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:55.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:56.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:57.842 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:57.843 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:57.843 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:57.843 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:57.844 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:57.844 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:57.844 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:57.844 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:57.845 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:57.845 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:57.845 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:57.845 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:57.845 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:57.845 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:57.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:58.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:59.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:00.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:01.845 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:01.846 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:01.846 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:01.846 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:01.847 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:01.847 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:01.847 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:01.847 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:01.847 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:01.848 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:01.848 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:01.848 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:01.848 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:01.848 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:01.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:02.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:03.336 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:05:03.344 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:05:03.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:04.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:05.847 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:05.850 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:05.850 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:05.851 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:05.851 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:05.851 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:05.852 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:05.853 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:05.853 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:05.853 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:05.854 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:05.854 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:05.854 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:05.855 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:07.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:08.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:09.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:09.851 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:09.853 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:09.854 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:09.854 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:09.854 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:09.855 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:09.855 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:09.856 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:09.856 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:09.856 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:09.856 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:09.856 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:09.857 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:09.857 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:11.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:12.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:13.015 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:13.855 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:13.857 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:13.857 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:13.857 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:13.858 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:13.858 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:13.858 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:13.859 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:13.859 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:13.860 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:13.860 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:13.860 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:13.860 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:13.861 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:15.969 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:16.179 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:05:16.186 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:05:16.971 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:17.857 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:17.859 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:17.859 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:17.859 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:17.859 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:17.859 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:17.859 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:17.859 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:17.859 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:17.860 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:17.860 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:17.860 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:17.860 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:17.860 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:17.972 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:18.974 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:19.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:20.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:21.860 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:21.862 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:21.863 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:21.863 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:21.863 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:21.864 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:21.864 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:21.865 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:21.865 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:21.865 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:21.866 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:21.866 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:21.866 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:21.866 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:21.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:22.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:23.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:24.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:25.863 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:25.938 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:25.939 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:25.939 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:25.939 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:25.939 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:25.939 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:25.940 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:25.940 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:25.940 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:25.940 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:25.940 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:25.940 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:25.940 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:25.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:26.542 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:05:26.543 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:05:26.543 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:05:26.543 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:05:26.544 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:05:26.544 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:05:26.544 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:05:26.544 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:05:26.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:27.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:28.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:29.112 7f7d9db39900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:05:29.118 7f7d9db39900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:05:29.940 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:29.941 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:29.941 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:29.941 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:29.941 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:29.942 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:29.942 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:29.942 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:29.942 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:29.942 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:29.942 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:29.942 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:29.942 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:29.942 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:29.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:30.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:31.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:33.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:33.943 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:33.946 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:33.946 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:33.947 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:33.947 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:33.948 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:33.948 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:33.949 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:33.949 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:33.950 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:33.950 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:33.950 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:33.951 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:33.951 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:35.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:36.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:37.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:37.947 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:37.949 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:37.949 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:37.950 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:37.950 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:37.950 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:37.950 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:37.951 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:37.951 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:37.952 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:37.952 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:37.952 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:37.952 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:37.952 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:39.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:40.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:41.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:41.950 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:41.951 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:41.952 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:41.952 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:41.952 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:41.953 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:41.953 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:41.954 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:41.954 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:41.954 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:41.954 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:41.955 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:41.955 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:41.955 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:42.045 7f7d9db39900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:05:42.054 7f7d9db39900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:05:43.019 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:44.021 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:45.952 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:45.955 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:45.955 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:45.955 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:45.956 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:45.956 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:45.956 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:45.957 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:45.957 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:45.958 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:45.958 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:45.958 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:45.958 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:45.958 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:45.969 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:46.971 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:47.972 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:48.973 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:49.955 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:49.957 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:49.958 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:49.958 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:49.958 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:49.958 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:49.959 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:49.959 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:49.960 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:49.960 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:49.960 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:49.960 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:49.961 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:49.961 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:49.974 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:50.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:51.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:52.979 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:53.958 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:53.960 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:53.961 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:53.961 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:53.961 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:53.962 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:53.962 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:53.963 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:53.963 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:53.964 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:53.964 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:53.964 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:53.964 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:53.964 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:53.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:54.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:55.217 7f7d9db39900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:05:55.223 7f7d9db39900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:05:55.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:56.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:57.962 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:57.964 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:57.965 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:57.965 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:57.965 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:57.966 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:57.966 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:57.967 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:57.967 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:57.967 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:57.967 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:57.968 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:57.968 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:57.968 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:57.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:58.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:59.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:00.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:01.965 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:01.966 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:01.966 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:01.966 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:01.967 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:01.967 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:01.967 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:01.967 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:01.967 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:01.967 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:01.967 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:01.967 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:01.967 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:01.967 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:01.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:02.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:04.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:05.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:05.968 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:05.971 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:05.971 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:05.971 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:05.972 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:05.972 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:05.972 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:05.973 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:05.973 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:05.974 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:05.974 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:05.974 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:05.974 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:05.975 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:07.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:08.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:08.214 7f7d9db39900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:06:08.222 7f7d9db39900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:06:09.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:09.972 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:09.975 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:09.975 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:09.976 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:09.976 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:09.976 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:09.977 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:09.977 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:09.978 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:09.978 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:09.978 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:09.979 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:09.979 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:09.979 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:11.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:12.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:13.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:13.975 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:13.976 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:13.976 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:13.976 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:13.976 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:13.976 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:13.976 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:13.977 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:13.977 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:13.977 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:13.977 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:13.977 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:13.977 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:13.977 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:15.970 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:16.972 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:17.974 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:17.977 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:17.979 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:17.979 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:17.979 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:17.980 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:17.980 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:17.980 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:17.981 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:17.981 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:17.982 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:17.982 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:17.982 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:17.982 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:17.982 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:19.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:20.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:21.218 7f7d9db39900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:06:21.227 7f7d9db39900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:06:21.979 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:21.980 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:21.983 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:21.983 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:21.983 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:21.984 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:21.984 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:21.985 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:21.986 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:21.986 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:21.987 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:21.987 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:21.987 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:21.988 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:21.988 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:23.983 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:24.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:25.984 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:25.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:25.987 7f7d9db39900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:25.988 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:25.988 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:25.988 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:25.989 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:25.989 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:25.990 7f7d9db39900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:25.990 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:25.991 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:25.991 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:25.991 7f7d9db39900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:25.991 7f7d9db39900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:25.992 7f7d9db39900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:27.609 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:06:27.609 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:06:27.609 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:06:27.610 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:06:27.610 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:06:27.610 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:06:27.610 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:06:27.611 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:06:27.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:28.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:29.990 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:29.992 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:29.993 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:29.993 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:29.994 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:29.994 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:29.994 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:29.995 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:29.995 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:29.996 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:29.996 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:29.996 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:29.996 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:29.996 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:29.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:30.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:32.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:33.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:33.983 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:06:33.991 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:06:33.993 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:33.995 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:33.995 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:33.995 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:33.996 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:33.996 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:33.996 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:33.997 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:33.997 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:33.997 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:33.997 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:33.998 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:33.998 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:33.998 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:35.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:36.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:37.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:37.995 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:37.998 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:37.998 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:37.998 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:37.999 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:37.999 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:37.999 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:38.000 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:38.000 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.001 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:38.001 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.001 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.001 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:38.001 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:39.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:40.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:41.015 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:41.998 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:42.001 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:42.002 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:42.002 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:42.003 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:42.004 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:42.004 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:42.006 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:42.006 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.007 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:42.007 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.008 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.008 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:42.008 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:43.018 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:44.019 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:45.971 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:46.001 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:46.003 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:46.003 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:46.003 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:46.003 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:46.003 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:46.003 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:46.004 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:46.004 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.004 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:46.004 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.004 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.004 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:46.004 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.869 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:06:46.878 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:06:46.973 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:47.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:48.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:49.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:50.003 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:50.004 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:50.004 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:50.005 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:50.005 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:50.005 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:50.005 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:50.005 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:50.005 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.005 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:50.005 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.005 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.005 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:50.006 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.979 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:51.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:52.983 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:53.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:54.005 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:54.006 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:54.006 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:54.006 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:54.006 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:54.007 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:54.007 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:54.007 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:54.007 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.007 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:54.007 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.007 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.007 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:54.007 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:55.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:56.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:57.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:58.007 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:58.009 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:58.009 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:58.010 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:58.010 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:58.010 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:58.011 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:58.011 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:58.012 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.012 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:58.012 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.012 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.012 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:58.013 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:59.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:00.047 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:07:00.051 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:07:00.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:02.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:02.010 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:02.012 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:02.013 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:02.013 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:02.014 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:02.014 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:02.015 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:02.015 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:02.016 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.016 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:02.016 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.016 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.017 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:02.017 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:03.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:04.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:05.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:06.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:06.013 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:06.016 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:06.016 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:06.016 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:06.017 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:06.017 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:06.017 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:06.018 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:06.018 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.019 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:06.019 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.019 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.020 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:06.020 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:07.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:08.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:09.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:10.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:10.017 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:10.020 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:10.020 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:10.020 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:10.020 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:10.021 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:10.021 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:10.022 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:10.022 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.022 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:10.022 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.023 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.023 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:10.023 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:11.018 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:12.020 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:13.022 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:13.076 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:07:13.087 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:07:14.020 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:14.021 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:14.022 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:14.022 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:14.022 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:14.022 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:14.022 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:14.023 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:14.023 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.023 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:14.023 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.024 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.024 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:14.024 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:15.972 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:16.974 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:17.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:18.023 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:18.025 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:18.025 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:18.026 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:18.026 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:18.026 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:18.026 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:18.027 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:18.027 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.028 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:18.028 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.028 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.028 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:18.029 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:19.979 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:20.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:21.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:22.025 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:22.028 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:22.028 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:22.029 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:22.029 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:22.030 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:22.030 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:22.031 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:22.031 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.032 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:22.032 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.032 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.033 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:22.033 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:23.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:24.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:25.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:26.029 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:26.031 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:26.031 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:26.031 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:26.032 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:26.032 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:26.032 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:26.033 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:26.033 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.034 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:26.034 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.034 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.034 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:26.034 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.179 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:07:26.186 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:07:26.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:27.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:28.543 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:07:28.543 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:07:28.544 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:07:28.544 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:07:28.545 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:07:28.545 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:07:28.545 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:07:28.546 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:07:28.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:29.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:30.031 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:30.033 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:30.034 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:30.034 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:30.034 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:30.035 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:30.035 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:30.036 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:30.036 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.036 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:30.036 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.037 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.037 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:30.037 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:31.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:32.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:33.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:34.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:34.034 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:34.035 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:34.035 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:34.035 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:34.035 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:34.035 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:34.035 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:34.036 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:34.036 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.036 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:34.036 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.036 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.036 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:34.036 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:35.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:36.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:37.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:38.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:38.036 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:38.038 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:38.038 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:38.038 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:38.039 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:38.039 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:38.039 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:38.040 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:38.040 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.041 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:38.041 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.041 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.041 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:38.042 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.918 7f7d9da835e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:07:38.924 7f7d9da835e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:07:39.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:40.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:41.019 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:42.021 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:42.038 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:42.041 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:42.041 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:42.041 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:42.042 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:42.042 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:42.042 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:42.043 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:42.043 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.044 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:42.044 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.044 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.044 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:42.044 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:43.023 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:44.025 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:45.974 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:46.042 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:46.045 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:46.045 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:46.045 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:46.046 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:46.046 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:46.047 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:46.047 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:46.048 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.048 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:46.048 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.048 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.049 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:46.049 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:47.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:48.979 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:49.981 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:50.045 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:50.047 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:50.047 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:50.047 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:50.047 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:50.047 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:50.048 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:50.048 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:50.048 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.049 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:50.049 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.049 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.049 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:50.049 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.983 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:51.669 7f7d9da835e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:07:51.675 7f7d9da835e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:07:51.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:52.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:53.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:54.047 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:54.049 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:54.050 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:54.050 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:54.050 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:54.050 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:54.051 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:54.051 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:54.052 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.052 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:54.052 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.052 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.053 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:54.053 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:55.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:56.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:57.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:58.050 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:58.051 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:58.051 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:58.052 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:58.052 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:58.052 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:58.052 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:58.052 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:58.053 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.053 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:58.053 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.053 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.053 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:58.053 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:00.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:01.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:02.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:02.052 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:02.053 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:02.053 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:02.053 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:02.054 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:02.054 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:02.054 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:02.054 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:02.054 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.054 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:02.054 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.054 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.054 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:02.055 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:03.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:04.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:04.691 7f7d9da835e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:04.697 7f7d9da835e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:05.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:06.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:06.054 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:06.056 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:06.057 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:06.057 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:06.057 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:06.058 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:06.058 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:06.059 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:06.059 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.059 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:06.059 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.060 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.060 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:06.060 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:07.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:08.018 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:09.020 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:10.022 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:10.057 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:10.058 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:10.059 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:10.059 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:10.059 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:10.059 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:10.059 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:10.059 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:10.059 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.060 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:10.060 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.060 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.060 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:10.060 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:11.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:12.026 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:13.028 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:14.030 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:14.060 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:14.062 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:14.062 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:14.063 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:14.063 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:14.063 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:14.064 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:14.064 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:14.065 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.065 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:14.065 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.066 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.066 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:14.066 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:15.974 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:16.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:17.808 7f7d9da835e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:17.819 7f7d9da835e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:17.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:18.063 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:18.065 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:18.066 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:18.066 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:18.067 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:18.067 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:18.067 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:18.068 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:18.068 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.068 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:18.069 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.069 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.069 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:18.070 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.979 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:19.981 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:20.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:21.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:22.066 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:22.067 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:22.067 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:22.067 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:22.067 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:22.067 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:22.067 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:22.068 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:22.068 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.068 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:22.068 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.068 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.068 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:22.068 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:23.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:24.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:25.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:26.067 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:26.070 7f7d9da835e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:26.070 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:26.070 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:26.071 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:26.071 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:26.071 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:26.072 7f7d9da835e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:26.073 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.073 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:26.073 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.073 7f7d9da835e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.074 7f7d9da835e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:26.074 7f7d9da835e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:27.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:28.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:29.508 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:08:29.509 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:08:29.509 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:08:29.509 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:08:29.510 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:08:29.510 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:08:29.510 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:08:29.510 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:08:29.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:30.070 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:30.071 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:30.071 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:30.071 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:30.072 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:30.072 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:30.072 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:30.072 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:30.072 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.072 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:30.072 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.072 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.072 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:30.073 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.565 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:30.572 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:31.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:32.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:33.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:34.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:34.072 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:34.073 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:34.073 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:34.073 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:34.073 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:34.073 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:34.074 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:34.074 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:34.074 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.074 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:34.074 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.074 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.074 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:34.074 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:35.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:36.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:37.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:38.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:38.074 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:38.077 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:38.077 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:38.077 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:38.078 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:38.078 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:38.078 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:38.079 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:38.080 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.080 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:38.080 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.080 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.081 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:38.081 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:39.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:40.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:41.019 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:42.021 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:42.078 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:42.080 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:42.080 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:42.081 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:42.081 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:42.082 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:42.082 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:42.083 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:42.083 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.084 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:42.084 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.084 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.084 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:42.085 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:43.023 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:43.234 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:43.239 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:44.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:45.975 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:46.081 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:46.083 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:46.084 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:46.084 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:46.085 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:46.085 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:46.085 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:46.086 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:46.086 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.086 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:46.087 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.087 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.087 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:46.087 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:47.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:48.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:49.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:50.084 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:50.085 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:50.086 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:50.086 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:50.086 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:50.086 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:50.086 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:50.086 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:50.087 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.087 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:50.087 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.087 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.087 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:50.087 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:51.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:52.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:53.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:54.086 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:54.088 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:54.089 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:54.089 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:54.089 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:54.090 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:54.090 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:54.091 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:54.091 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.091 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:54.091 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.091 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.092 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:54.092 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:55.895 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:55.905 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:55.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:56.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:57.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:58.089 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:58.091 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:58.092 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:58.092 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:58.092 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:58.093 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:58.093 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:58.094 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:58.094 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.095 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:58.095 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.095 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.096 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:58.096 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:00.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:01.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:02.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:02.092 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:02.095 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:02.096 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:02.096 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:02.096 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:02.097 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:02.097 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:02.098 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:02.098 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.098 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:02.099 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.099 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.099 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:02.099 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:03.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:04.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:05.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:06.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:06.096 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:06.097 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:06.098 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:06.098 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:06.098 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:06.098 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:06.098 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:06.098 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:06.099 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.099 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:06.099 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.099 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.099 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:06.099 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:07.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:08.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:09.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:09.024 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:09:09.032 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:09:10.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:10.098 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:10.100 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:10.101 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:10.101 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:10.102 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:10.102 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:10.102 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:10.103 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:10.104 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.104 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:10.104 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.105 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.105 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:10.105 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:11.018 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:12.020 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:13.022 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:14.023 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:14.101 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:14.103 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:14.104 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:14.104 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:14.104 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:14.105 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:14.105 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:14.106 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:14.106 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.106 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:14.106 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.107 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.107 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:14.107 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:15.976 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:16.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:17.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:18.104 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:18.106 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:18.107 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:18.107 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:18.107 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:18.108 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:18.108 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:18.109 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:18.109 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.109 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:18.110 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.110 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.110 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:18.110 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:19.983 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:20.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:21.828 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:09:21.834 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:09:21.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:22.107 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:22.109 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:22.110 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:22.110 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:22.111 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:22.111 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:22.111 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:22.112 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:22.112 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.112 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:22.114 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.114 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.115 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:22.115 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:23.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:24.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:25.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:26.109 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:26.111 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:26.111 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:26.112 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:26.112 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:26.112 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:26.113 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:26.114 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:26.114 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.114 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:26.114 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.115 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.115 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:26.115 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:28.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:29.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:30.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:30.112 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:30.114 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:30.114 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:30.115 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:30.115 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:30.115 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:30.115 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:30.116 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:30.116 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.117 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:30.117 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.117 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.117 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:30.117 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.519 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:09:30.520 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:09:30.520 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:09:30.520 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:09:30.521 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:09:30.521 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:09:30.521 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:09:30.521 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:09:31.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:32.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:33.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:34.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:34.115 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:34.118 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:34.118 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:34.118 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:34.119 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:34.119 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:34.120 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:34.121 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:34.121 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.121 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:34.122 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.122 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.122 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:34.122 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.770 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:09:34.776 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:09:35.015 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:36.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:37.018 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:38.019 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:38.118 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:38.121 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:38.121 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:38.122 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:38.122 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:38.122 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:38.122 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:38.123 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:38.124 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.124 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:38.124 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.125 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.125 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:38.125 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:39.022 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:40.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:41.026 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:42.028 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:42.122 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:42.124 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:42.124 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:42.124 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:42.125 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:42.125 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:42.125 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:42.126 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:42.126 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.127 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:42.127 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.127 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.127 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:42.128 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:43.030 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:44.032 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:45.977 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:46.124 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:46.127 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:46.127 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:46.128 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:46.128 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:46.128 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:46.128 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:46.129 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:46.130 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.130 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:46.131 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.131 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.131 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:46.131 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:47.756 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:09:47.764 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:09:47.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:48.983 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:49.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:50.128 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:50.130 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:50.130 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:50.131 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:50.131 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:50.131 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:50.131 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:50.132 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:50.132 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.133 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:50.133 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.133 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.134 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:50.134 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:51.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:52.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:53.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:54.130 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:54.133 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:54.133 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:54.134 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:54.134 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:54.135 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:54.135 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:54.136 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:54.136 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.137 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:54.137 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.138 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.138 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:54.138 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:55.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:56.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:57.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:58.134 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:58.135 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:58.135 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:58.135 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:58.135 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:58.135 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:58.135 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:58.136 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:58.136 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.136 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:58.136 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.136 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.136 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:58.136 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:59.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:00.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:00.331 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:10:00.335 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:10:01.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:02.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:02.136 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:02.138 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:02.139 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:02.139 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:02.140 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:02.140 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:02.140 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:02.141 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:02.142 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.142 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:02.142 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.142 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.143 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:02.143 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:03.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:04.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:05.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:06.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:06.139 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:06.141 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:06.141 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:06.141 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:06.141 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:06.141 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:06.142 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:06.142 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:06.142 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.142 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:06.143 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.143 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.143 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:06.143 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:07.015 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:08.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:09.018 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:10.020 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:10.141 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:10.143 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:10.144 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:10.144 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:10.145 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:10.145 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:10.145 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:10.146 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:10.146 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.146 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:10.147 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.147 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.147 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:10.147 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:11.022 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:12.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:13.025 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:13.332 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:10:13.341 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:10:14.026 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:14.145 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:14.147 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:14.148 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:14.148 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:14.148 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:14.149 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:14.149 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:14.150 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:14.150 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.151 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:14.151 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.151 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.151 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:14.151 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:15.978 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:16.979 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:17.981 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:18.148 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:18.150 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:18.150 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:18.151 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:18.151 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:18.151 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:18.152 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:18.152 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:18.153 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.153 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:18.153 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.153 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.154 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:18.154 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:19.983 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:20.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:21.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:22.151 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:22.153 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:22.154 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:22.154 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:22.154 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:22.154 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:22.154 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:22.155 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:22.156 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.156 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:22.156 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.157 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.157 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:22.157 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:23.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:24.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:25.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:26.089 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:10:26.097 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:10:26.154 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:26.156 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:26.157 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:26.157 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:26.158 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:26.158 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:26.158 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:26.159 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:26.159 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.159 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:26.160 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.160 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.160 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:26.160 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:27.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:28.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:30.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:30.157 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:30.159 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:30.159 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:30.160 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:30.160 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:30.160 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:30.160 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:30.161 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:30.161 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.161 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:30.161 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.161 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.161 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:30.162 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:31.389 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:10:31.389 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:10:31.389 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:10:31.389 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:10:31.390 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:10:31.390 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:10:31.390 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:10:31.390 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:10:32.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:33.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:34.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:34.159 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:34.161 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:34.161 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:34.161 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:34.161 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:34.161 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:34.161 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:34.162 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:34.162 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.162 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:34.162 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.162 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.162 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:34.162 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:36.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:37.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:38.015 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:38.162 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:38.164 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:38.164 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:38.164 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:38.164 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:38.165 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:38.165 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:38.165 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:38.165 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.165 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:38.166 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.166 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.166 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:38.166 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.018 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:39.115 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:10:39.122 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:10:40.020 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:41.022 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:42.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:42.164 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:42.167 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:42.167 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:42.167 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:42.168 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:42.168 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:42.168 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:42.169 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:42.169 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.170 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:42.170 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.170 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.170 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:42.171 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.025 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:44.027 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:45.979 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:46.168 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:46.170 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:46.171 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:46.171 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:46.172 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:46.172 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:46.172 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:46.173 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:46.173 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:46.174 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:46.174 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:46.174 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:46.175 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:46.175 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:46.981 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:47.983 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:48.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:49.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:50.171 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:50.174 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:50.174 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:50.175 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:50.175 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:50.175 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:50.175 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:50.176 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:50.176 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:50.177 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:50.177 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:50.177 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:50.178 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:50.178 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:50.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:51.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:52.057 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:10:52.068 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:10:52.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:53.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:54.175 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:54.177 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:54.177 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:54.177 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:54.177 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:54.178 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:54.178 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:54.178 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:54.178 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.179 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:54.179 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.179 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.179 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:54.179 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:55.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:56.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:58.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:58.178 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:58.181 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:58.181 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:58.182 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:58.182 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:58.182 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:58.182 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:58.183 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:58.183 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:58.184 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:58.184 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:58.184 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:58.185 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:58.185 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:00.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:01.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:02.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:02.181 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:02.183 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:02.184 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:02.184 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:02.185 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:02.185 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:02.185 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:02.186 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:02.186 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:02.187 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:02.187 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:02.187 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:02.187 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:02.188 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:04.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:04.866 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:04.875 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:05.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:06.019 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:06.184 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:06.186 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:06.186 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:06.187 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:06.187 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:06.187 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:06.188 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:06.188 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:06.189 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:06.189 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:06.189 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:06.190 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:06.190 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:06.190 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.020 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:08.022 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:09.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:10.027 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:10.188 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:10.190 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:10.190 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:10.191 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:10.191 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:10.192 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:10.192 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:10.193 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:10.194 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:10.194 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:10.195 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:10.195 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:10.195 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:10.196 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.029 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:12.031 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:13.033 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:14.035 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:14.192 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:14.194 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:14.194 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:14.194 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:14.195 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:14.195 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:14.196 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:14.196 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:14.197 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.197 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:14.197 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.197 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.198 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:14.198 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.980 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:16.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:17.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:18.094 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:18.102 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:18.194 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:18.197 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:18.197 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:18.197 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:18.198 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:18.198 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:18.198 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:18.199 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:18.199 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:18.200 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:18.200 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:18.200 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:18.200 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:18.200 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:18.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:19.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:20.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:21.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:22.197 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:22.198 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:22.199 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:22.199 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:22.199 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:22.199 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:22.199 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:22.199 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:22.200 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:22.200 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:22.200 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:22.200 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:22.200 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:22.200 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:22.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:23.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:24.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:26.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:26.200 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:26.202 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:26.202 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:26.203 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:26.203 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:26.203 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:26.203 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:26.204 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:26.205 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:26.205 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:26.206 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:26.207 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:26.207 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:26.207 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:28.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:29.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:30.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:30.203 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:30.206 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:30.206 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:30.206 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:30.207 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:30.207 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:30.207 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:30.208 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:30.208 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:30.209 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:30.209 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:30.210 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:30.210 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:30.210 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:30.988 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:30.994 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:31.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:32.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:32.444 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:11:32.445 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:11:32.445 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:11:32.445 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:11:32.445 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:11:32.445 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:11:32.446 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:11:32.446 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:11:33.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:34.018 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:34.206 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:34.209 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:34.209 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:34.209 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:34.210 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:34.210 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:34.210 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:34.211 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:34.211 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.211 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:34.212 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.212 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.212 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:34.212 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.020 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:36.021 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:37.023 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:38.025 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:38.210 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:38.212 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:38.212 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:38.213 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:38.213 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:38.213 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:38.214 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:38.214 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:38.215 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:38.215 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:38.215 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:38.215 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:38.216 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:38.216 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.029 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:40.031 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:41.032 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:42.033 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:42.213 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:42.216 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:42.216 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:42.217 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:42.217 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:42.217 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:42.218 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:42.219 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:42.219 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:42.219 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:42.220 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:42.220 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:42.220 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:42.221 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.035 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:43.905 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:43.914 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:44.037 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:45.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:46.217 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:46.219 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:46.219 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:46.220 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:46.220 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:46.220 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:46.220 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:46.221 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:46.221 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:46.222 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:46.222 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:46.222 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:46.222 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:46.223 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:46.983 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:47.985 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:48.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:49.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:50.220 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:50.223 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:50.223 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:50.223 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:50.224 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:50.224 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:50.224 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:50.225 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:50.225 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:50.226 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:50.226 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:50.226 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:50.226 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:50.226 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:50.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:51.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:52.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:53.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:54.223 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:54.225 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:54.226 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:54.226 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:54.227 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:54.227 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:54.227 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:54.228 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:54.228 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.229 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:54.229 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.229 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.230 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:54.230 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:56.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:56.880 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:56.885 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:57.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:58.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:58.226 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:58.228 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:58.229 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:58.229 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:58.229 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:58.230 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:58.230 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:58.231 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:58.231 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:58.231 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:58.231 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:58.231 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:58.232 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:58.232 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:00.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:01.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:02.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:02.229 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:02.230 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:02.230 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:02.230 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:02.231 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:02.231 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:02.231 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:02.232 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:02.232 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:02.232 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:02.232 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:02.232 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:02.232 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:02.232 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:04.015 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:05.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:06.018 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:06.231 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:06.232 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:06.233 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:06.233 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:06.233 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:06.233 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:06.233 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:06.233 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:06.233 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:06.233 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:06.233 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:06.233 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:06.233 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:06.233 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.021 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:08.022 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:09.025 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:09.755 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:12:09.759 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:12:10.026 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:10.233 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:10.235 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:10.236 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:10.236 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:10.237 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:10.237 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:10.237 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:10.239 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:10.239 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:10.240 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:10.240 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:10.240 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:10.241 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:10.241 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.028 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:12.030 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:13.031 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:14.033 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:14.236 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:14.239 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:14.239 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:14.240 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:14.240 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:14.240 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:14.240 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:14.241 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:14.242 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.242 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:14.242 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.242 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.243 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:14.243 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.982 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:16.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:17.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:18.240 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:18.241 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:18.242 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:18.242 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:18.242 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:18.242 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:18.242 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:18.242 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:18.242 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:18.242 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:18.243 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:18.243 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:18.243 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:18.243 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:18.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:19.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:20.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:21.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:22.243 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:22.245 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:22.246 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:22.246 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:22.246 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:22.247 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:22.247 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:22.248 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:22.248 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:22.248 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:22.249 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:22.249 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:22.249 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:22.249 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:22.893 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:12:22.900 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:12:22.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:23.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:24.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:25.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:26.246 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:26.248 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:26.248 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:26.248 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:26.249 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:26.249 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:26.249 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:26.249 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:26.250 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:26.250 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:26.250 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:26.250 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:26.250 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:26.250 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:28.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:29.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:30.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:30.249 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:30.252 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:30.252 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:30.252 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:30.253 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:30.253 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:30.253 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:30.254 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:30.254 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:30.255 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:30.255 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:30.255 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:30.255 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:30.256 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:32.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:33.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:33.537 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:12:33.538 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:12:33.538 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:12:33.538 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:12:33.538 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:12:33.538 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:12:33.539 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:12:33.539 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:12:34.015 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:34.252 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:34.255 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:34.255 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:34.255 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:34.256 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:34.256 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:34.256 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:34.257 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:34.257 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.258 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:34.258 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.258 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.258 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:34.259 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.020 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:35.651 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:12:35.659 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:12:36.022 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:37.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:38.025 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:38.255 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:38.258 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:38.258 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:38.259 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:38.259 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:38.259 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:38.260 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:38.261 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:38.261 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:38.262 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:38.262 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:38.262 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:38.263 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:38.263 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.026 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:40.028 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:41.030 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:42.032 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:42.259 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:42.260 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:42.260 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:42.260 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:42.260 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:42.260 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:42.260 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:42.261 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:42.261 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:42.261 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:42.261 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:42.261 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:42.261 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:42.261 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.034 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:44.035 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:45.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:46.261 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:46.263 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:46.264 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:46.264 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:46.264 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:46.264 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:46.265 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:46.266 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:46.266 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:46.266 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:46.266 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:46.267 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:46.267 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:46.267 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:46.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:47.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:48.682 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:12:48.690 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:12:48.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:49.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:50.264 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:50.265 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:50.265 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:50.265 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:50.265 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:50.266 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:50.266 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:50.266 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:50.266 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:50.266 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:50.266 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:50.266 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:50.266 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:50.266 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:50.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:51.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:52.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:53.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:54.266 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:54.268 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:54.268 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:54.268 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:54.269 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:54.269 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:54.269 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:54.270 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:54.270 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:54.271 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:54.271 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:54.271 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:54.271 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:54.271 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:56.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:57.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:58.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:58.269 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:58.271 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:58.271 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:58.272 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:58.272 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:58.272 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:58.273 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:58.274 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:58.274 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:58.275 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:58.275 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:58.275 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:58.276 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:58.276 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:00.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:01.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:01.610 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:01.615 7f7d9da83040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:13:02.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:02.272 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:02.274 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:02.275 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:02.276 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:02.276 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:02.276 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:02.277 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:02.277 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:02.278 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:02.278 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:02.278 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:02.279 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:02.279 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:02.279 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:04.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:05.019 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:06.021 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:06.275 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:06.277 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:06.278 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:06.278 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:06.278 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:06.278 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:06.279 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:06.279 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:06.279 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:06.280 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:06.280 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:06.280 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:06.280 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:06.280 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.023 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:08.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:09.026 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:10.028 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:10.278 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:10.279 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:10.279 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:10.279 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:10.280 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:10.280 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:10.280 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:10.280 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:10.280 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:10.280 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:10.280 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:10.280 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:10.281 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:10.281 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.029 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:12.032 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:13.034 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:14.035 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:14.281 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:14.283 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:14.284 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:14.284 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:14.285 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:14.285 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:14.286 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:14.287 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:14.287 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.288 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:14.288 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.289 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.289 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:14.290 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.493 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:14.503 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:13:15.984 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:16.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:17.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:18.285 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:18.287 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:18.287 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:18.287 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:18.288 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:18.288 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:18.288 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:18.289 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:18.289 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:18.290 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:18.290 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:18.290 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:18.290 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:18.291 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:18.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:19.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:20.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:21.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:22.287 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:22.288 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:22.288 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:22.288 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:22.289 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:22.289 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:22.289 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:22.289 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:22.289 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:22.290 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:22.290 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:22.290 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:22.290 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:22.290 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:22.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:23.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:25.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:26.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:26.289 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:26.291 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:26.292 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:26.292 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:26.292 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:26.293 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:26.293 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:26.294 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:26.294 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:26.295 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:26.295 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:26.295 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:26.295 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:26.296 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:27.849 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:27.857 7f7d9db69c20 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:13:28.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:29.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:30.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:30.293 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:30.294 7f7d9db69c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:30.294 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:30.294 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:30.294 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:30.294 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:30.294 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:30.294 7f7d9db69c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:30.295 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:30.295 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:30.295 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:30.295 7f7d9db69c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:30.295 7f7d9db69c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:30.295 7f7d9db69c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:32.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:33.015 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:34.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:34.295 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:34.297 7f7d9da83040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:34.298 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:34.298 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:34.298 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:34.299 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:34.299 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:34.300 7f7d9da83040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:34.300 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.300 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:34.301 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.301 7f7d9da83040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.301 7f7d9da83040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:34.301 7f7d9da83040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.528 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:13:34.528 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:13:34.529 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:13:34.529 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:13:34.529 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:13:34.529 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:13:34.529 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:13:34.529 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:13:35.019 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:36.021 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:37.023 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:38.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:38.299 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:38.301 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:38.302 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:38.302 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:38.302 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:38.302 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:38.303 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:38.303 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:38.304 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:38.304 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:38.304 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:38.304 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:38.305 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:38.305 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.027 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:40.029 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:40.718 7f7d9da837c0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:40.724 7f7d9da837c0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:13:41.030 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:42.033 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:42.301 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:42.303 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:42.304 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:42.304 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:42.304 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:42.305 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:42.305 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:42.306 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:42.306 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:42.306 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:42.306 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:42.306 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:42.307 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:42.307 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.035 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:44.037 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:45.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:46.304 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:46.306 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:46.307 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:46.307 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:46.307 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:46.308 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:46.308 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:46.309 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:46.309 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:46.309 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:46.309 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:46.310 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:46.310 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:46.310 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:46.987 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:47.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:48.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:49.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:50.307 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:50.310 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:50.310 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:50.310 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:50.311 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:50.311 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:50.311 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:50.312 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:50.312 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:50.313 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:50.313 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:50.313 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:50.313 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:50.313 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:50.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:51.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:52.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:53.457 7f7d9da837c0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:53.461 7f7d9da837c0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:13:53.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:54.310 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:54.312 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:54.313 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:54.313 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:54.314 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:54.314 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:54.314 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:54.315 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:54.315 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:54.316 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:54.316 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:54.316 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:54.316 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:54.317 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:56.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:57.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:58.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:58.313 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:58.314 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:58.314 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:58.314 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:58.314 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:58.315 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:58.315 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:58.315 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:58.315 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:58.315 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:58.315 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:58.315 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:58.315 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:58.315 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:00.009 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:01.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:02.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:02.315 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:02.318 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:02.319 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:02.319 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:02.319 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:02.319 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:02.320 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:02.320 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:02.321 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:02.321 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:02.321 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:02.322 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:02.322 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:02.322 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:04.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:05.019 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:06.021 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:06.178 7f7d9da837c0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:06.189 7f7d9da837c0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:06.319 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:06.322 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:06.322 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:06.322 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:06.323 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:06.323 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:06.323 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:06.324 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:06.324 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:06.325 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:06.325 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:06.325 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:06.326 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:06.326 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.023 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:08.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:09.026 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:10.028 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:10.322 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:10.325 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:10.325 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:10.325 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:10.326 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:10.326 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:10.326 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:10.327 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:10.327 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:10.328 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:10.328 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:10.328 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:10.328 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:10.328 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.030 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:12.032 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:13.034 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:14.035 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:14.325 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:14.327 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:14.327 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:14.327 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:14.327 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:14.327 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:14.328 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:14.328 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:14.328 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.328 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:14.328 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.328 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.328 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:14.328 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.986 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:16.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:17.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:18.328 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:18.330 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:18.330 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:18.330 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:18.330 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:18.330 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:18.331 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:18.331 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:18.331 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:18.331 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:18.332 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:18.332 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:18.332 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:18.332 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:18.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:19.068 7f7d9da837c0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:19.075 7f7d9da837c0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:19.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:20.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:21.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:22.331 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:22.333 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:22.334 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:22.334 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:22.335 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:22.335 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:22.335 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:22.337 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:22.337 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:22.338 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:22.338 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:22.338 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:22.338 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:22.339 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:24.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:25.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:26.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:26.334 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:26.335 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:26.335 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:26.335 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:26.336 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:26.336 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:26.336 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:26.336 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:26.336 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:26.336 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:26.337 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:26.337 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:26.337 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:26.337 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:28.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:29.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:30.015 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:30.336 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:30.338 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:30.338 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:30.338 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:30.338 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:30.338 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:30.338 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:30.338 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:30.339 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:30.339 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:30.339 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:30.339 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:30.339 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:30.339 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:32.018 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:32.133 7f7d9da837c0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:32.144 7f7d9da837c0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:33.021 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:34.023 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:34.339 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:34.341 7f7d9da837c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:34.342 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:34.342 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:34.342 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:34.342 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:34.343 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:34.343 7f7d9da837c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:34.344 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:34.344 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:34.344 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:34.344 7f7d9da837c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:34.345 7f7d9da837c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:34.345 7f7d9da837c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:35.542 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:14:35.542 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:14:35.543 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:14:35.543 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:14:35.543 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:14:35.543 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:14:35.544 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:14:35.544 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:14:36.026 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:37.028 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:38.029 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:38.342 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:38.345 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:38.345 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:38.346 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:38.346 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:38.346 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:38.347 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:38.348 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:38.348 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:38.348 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:38.349 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:38.349 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:38.349 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:38.350 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.031 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:40.033 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:41.034 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:42.036 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:42.346 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:42.349 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:42.349 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:42.349 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:42.350 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:42.350 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:42.350 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:42.351 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:42.352 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:42.352 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:42.352 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:42.352 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:42.353 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:42.353 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.038 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:44.040 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:44.988 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:44.998 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:45.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:46.350 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:46.352 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:46.352 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:46.352 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:46.352 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:46.352 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:46.352 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:46.353 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:46.353 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:46.353 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:46.353 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:46.353 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:46.353 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:46.353 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:46.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:47.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:48.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:49.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:50.353 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:50.355 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:50.356 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:50.356 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:50.356 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:50.357 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:50.357 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:50.358 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:50.358 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:50.358 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:50.358 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:50.359 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:50.359 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:50.359 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:50.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:51.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:53.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:54.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:54.357 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:54.359 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:54.359 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:54.359 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:54.360 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:54.360 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:54.361 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:54.362 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:54.362 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:54.362 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:54.362 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:54.363 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:54.363 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:54.363 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.005 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:56.007 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:57.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:57.952 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:57.960 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:58.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:58.360 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:58.362 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:58.362 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:58.362 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:58.363 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:58.363 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:58.363 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:58.363 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:58.364 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:58.364 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:58.364 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:58.364 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:58.364 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:58.364 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:59.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:00.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:01.015 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:02.017 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:02.363 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:02.364 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:02.364 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:02.364 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:02.364 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:02.364 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:02.365 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:02.365 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:02.365 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:02.365 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:02.365 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:02.365 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:02.365 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:02.366 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:03.019 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:04.020 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:05.022 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:06.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:06.366 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:06.368 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:06.368 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:06.369 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:06.369 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:06.370 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:06.370 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:06.371 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:06.371 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:06.371 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:06.372 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:06.372 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:06.372 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:06.372 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:07.027 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:08.028 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:09.030 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:10.032 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:10.369 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:10.371 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:10.372 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:10.372 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:10.372 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:10.372 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:10.372 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:10.373 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:10.373 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:10.373 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:10.373 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:10.374 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:10.374 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:10.374 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:11.019 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:15:11.030 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:15:11.033 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:12.035 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:13.037 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:14.038 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:14.373 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:14.375 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:14.375 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:14.376 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:14.376 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:14.376 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:14.376 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:14.377 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:14.378 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:14.378 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:14.378 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:14.379 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:14.379 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:14.379 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:15.988 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:16.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:17.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:18.376 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:18.378 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:18.379 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:18.379 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:18.379 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:18.380 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:18.380 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:18.381 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:18.381 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:18.381 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:18.382 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:18.382 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:18.382 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:18.382 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:18.993 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:19.995 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:20.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:21.998 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:22.379 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:22.381 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:22.382 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:22.382 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:22.382 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:22.382 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:22.383 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:22.383 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:22.384 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:22.384 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:22.384 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:22.384 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:22.385 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:22.385 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:23.000 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:23.842 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:15:23.847 7f7d9da83360 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:15:24.002 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:25.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:26.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:26.382 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:26.385 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:26.385 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:26.386 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:26.387 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:26.387 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:26.387 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:26.389 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:26.389 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:26.390 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:26.390 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:26.390 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:26.391 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:26.391 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:27.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:28.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:29.011 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:30.014 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:30.386 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:30.388 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:30.388 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:30.389 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:30.389 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:30.389 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:30.390 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:30.390 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:30.391 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:30.391 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:30.391 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:30.391 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:30.392 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:30.392 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:31.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:32.018 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:33.019 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:34.022 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:34.389 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:34.391 7f7d9da83360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:34.392 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:34.392 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:34.392 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:34.392 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:34.393 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:34.394 7f7d9da83360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:34.394 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:34.394 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:34.394 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:34.394 7f7d9da83360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:34.395 7f7d9da83360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:34.395 7f7d9da83360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:35.023 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:36.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:36.423 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:15:36.423 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:15:36.424 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:15:36.424 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:15:36.424 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:15:36.424 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:15:36.424 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:15:36.424 7f7d9ddbd180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:15:37.026 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:37.105 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:15:37.114 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:15:38.027 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:38.393 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:38.395 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:38.396 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:38.396 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:38.396 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:38.397 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:38.397 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:38.398 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:38.400 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:38.400 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:38.400 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:38.400 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:38.401 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:38.401 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:39.029 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:39.864 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:15:39.872 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:15:40.030 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:41.032 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:42.034 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:42.396 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:42.398 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:42.399 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:42.399 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:42.399 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:42.400 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:42.400 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:42.401 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:42.401 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:42.402 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:42.402 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:42.402 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:42.403 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:42.403 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:42.768 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:15:42.776 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:15:43.036 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:44.037 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:45.990 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:46.399 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:46.400 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:46.400 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:46.400 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:46.400 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:46.400 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:46.400 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:46.401 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:46.401 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:46.401 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:46.401 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:46.401 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:46.401 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:46.401 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:46.992 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:47.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:48.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:49.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:50.401 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:50.402 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:50.402 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:50.402 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:50.402 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:50.402 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:50.402 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:50.403 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:50.403 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:50.403 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:50.403 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:50.403 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:50.403 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:50.403 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:50.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:52.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:53.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:54.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:54.403 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:54.406 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:54.406 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:54.406 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:54.406 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:54.406 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:54.407 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:54.407 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:54.407 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:54.407 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:54.408 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:54.408 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:54.408 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:54.408 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:55.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:56.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:57.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:58.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:58.407 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:58.409 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:58.410 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:58.410 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:58.410 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:58.411 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:58.411 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:58.412 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:58.412 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:58.412 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:58.412 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:58.413 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:58.413 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:58.413 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:59.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:00.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:01.018 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:02.019 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:02.410 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:02.411 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:02.411 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:02.411 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:02.411 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:02.412 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:02.412 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:02.412 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:02.412 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:02.412 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:02.412 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:02.412 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:02.412 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:02.412 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:03.021 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:04.024 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:05.025 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:06.027 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:06.413 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:06.415 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:06.416 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:06.416 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:06.416 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:06.417 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:06.417 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:06.418 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:06.418 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:06.418 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:06.418 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:06.419 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:06.419 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:06.419 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:07.030 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:08.031 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:09.034 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:10.036 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:10.416 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:10.417 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:10.417 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:10.417 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:10.417 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:10.417 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:10.417 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:10.418 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:10.418 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:10.418 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:10.418 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:10.418 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:10.418 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:10.418 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:11.038 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:12.040 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:13.042 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:14.044 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:14.418 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:14.420 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:14.420 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:14.421 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:14.421 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:14.421 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:14.422 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:14.422 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:14.423 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:14.423 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:14.423 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:14.423 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:14.424 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:14.424 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:15.989 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:16.991 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:17.119 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:16:17.124 7f7d9da83900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:16:17.994 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:18.421 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:18.423 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:18.424 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:18.424 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:18.424 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:18.425 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:18.425 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:18.426 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:18.426 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:18.426 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:18.427 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:18.427 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:18.427 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:18.427 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:18.996 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:19.997 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:20.999 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:22.001 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:22.424 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:22.426 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:22.426 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:22.426 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:22.426 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:22.426 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:22.427 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:22.427 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:22.427 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:22.427 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:22.428 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:22.428 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:22.428 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:22.428 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:23.003 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:24.004 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:25.006 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:26.008 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:26.427 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:26.429 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:26.429 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:26.430 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:26.430 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:26.430 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:26.430 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:26.431 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:26.431 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:26.432 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:26.432 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:26.432 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:26.432 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:26.433 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:27.010 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:28.012 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:29.013 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:30.016 7f7d9d927ae0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:30.430 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:30.432 7f7d9da83900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:30.432 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:30.432 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:30.432 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:30.433 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:30.433 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:30.434 7f7d9da83900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:30.434 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:30.434 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:30.434 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:30.435 7f7d9da83900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:30.435 7f7d9da83900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691769348178 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:30.435 7f7d9da83900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]