09/01/2025 01:57:15.791 7f75df5eec80 [contrail-api] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/01/2025 01:57:15.791 7f75df5eec80 [contrail-api] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/01/2025 01:57:15.791 7f75df5eec80 [contrail-api] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/01/2025 01:57:15.792 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to zookeeper on 10.0.0.56:2181 09/01/2025 01:57:15.792 7f75df5eec80 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 01:57:15.793 7f75db8f3900 [contrail-api] [ERROR]: Cannot write http_port 8084 to /tmp/contrail-api.0.http_port 09/01/2025 01:57:15.793 7f75db8f3900 [contrail-api] [ERROR]: Starting Introspect on HTTP Port 8084 09/01/2025 01:57:15.793 7f75db8f3cc0 [contrail-api] [INFO]: Processing event[EvStart] in state[none] 09/01/2025 01:57:15.793 7f75db8f3cc0 [contrail-api] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 01:57:15.793 7f75db8f3cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/01/2025 01:57:15.793 7f75db8f3cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.793 7f75db8f3cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 01:57:15.793 7f75db8f3cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.794 7f75db8f3cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.794 7f75db8f3cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:15.794 7f75db8f3cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.794 7f75db8f3cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.794 7f75db8f3cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 09/01/2025 01:57:15.794 7f75db8f3cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.798 7f75db8f3cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.798 7f75db8f3cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:15.798 7f75db8f3cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.846 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to database on ['10.0.0.56:9041'] 09/01/2025 01:57:15.846 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: VNCCassandra started with driver cql 09/01/2025 01:57:15.846 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: initializing cassandra cluster 09/01/2025 01:57:15.846 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster before create 09/01/2025 01:57:15.847 7f75db8f3d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.847 7f75db8f3d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:57:15.847 7f75db8f3d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.849 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster created 09/01/2025 01:57:15.896 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 01:57:15.897 7f75dad1eea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.897 7f75dad1eea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:57:15.897 7f75dad1eea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.901 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 01:57:15.902 7f75dad1ee00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.902 7f75dad1ee00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:57:15.902 7f75dad1ee00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.913 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:57:15.984 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'config_db_uuid', created 09/01/2025 01:57:15.988 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:57:15.993 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:57:16.057 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'config_db_uuid' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 09/01/2025 01:57:16.062 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:57:16.073 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:57:16.148 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'useragent', created 09/01/2025 01:57:16.152 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:57:16.159 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:57:16.217 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'useragent' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 09/01/2025 01:57:16.223 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:57:16.223 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 01:57:16.243 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 01:57:16.250 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:57:16.436 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_uuid_table', created 09/01/2025 01:57:16.442 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:57:16.451 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:57:16.564 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 01:57:16.568 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:57:16.574 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:57:16.745 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_fq_name_table', created 09/01/2025 01:57:16.752 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:57:16.977 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:57:17.116 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 01:57:17.120 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:57:17.237 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:57:17.414 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_shared_table', created 09/01/2025 01:57:17.419 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:57:17.605 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:57:17.722 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 01:57:17.727 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:57:18.063 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:57:18.281 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'useragent_keyval_table', created 09/01/2025 01:57:18.288 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:57:18.570 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:57:18.717 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'useragent_keyval_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 01:57:18.725 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:57:18.726 7f75db7edae0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.726 7f75db7edae0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:18.726 7f75db7edae0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.823 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.56:5673//'] 09/01/2025 01:57:18.830 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection down 09/01/2025 01:57:18.831 7f75dacacc20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.831 7f75dacacc20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:57:18.831 7f75dacacc20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.831 7f75dacacc20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.831 7f75dacacc20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:57:18.831 7f75dacacc20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.834 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:57:18.835 7f75db801360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.835 7f75db801360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:18.835 7f75db801360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.845 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/01/2025 01:57:18.845 7f75daccb540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.846 7f75daccb540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:57:18.846 7f75daccb540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.850 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:57:18.851 7f75daccbc20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.851 7f75daccbc20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:18.851 7f75daccbc20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.852 7f75daccbea0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/01/2025 01:57:18.854 7f75daccb860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.854 7f75daccb860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:57:18.854 7f75daccb860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.858 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config']: Unknown id: global_system_config default-global-system-config 09/01/2025 01:57:18.859 7f75daccbea0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:57:18.860 7f75daccbb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.860 7f75daccbb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:18.860 7f75daccbb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.894 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-global-vrouter-config']: Unknown id: global_vrouter_config default-global-system-config:default-global-vrouter-config 09/01/2025 01:57:18.931 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain']: Unknown id: domain default-domain 09/01/2025 01:57:19.054 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-fabric']: Unknown id: fabric default-global-system-config:default-fabric 09/01/2025 01:57:19.079 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-policy-management']: Unknown id: policy_management default-policy-management 09/01/2025 01:57:19.099 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-policy-management', 'default-application-policy-set']: Unknown id: application_policy_set default-policy-management:default-application-policy-set 09/01/2025 01:57:19.140 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'ip-fabric']: Unknown id: virtual_network default-domain:default-project:ip-fabric 09/01/2025 01:57:19.221 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric']: Unknown id: routing_instance default-domain:default-project:ip-fabric:ip-fabric 09/01/2025 01:57:19.247 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'ip-fabric', '__default__']: Unknown id: routing_instance default-domain:default-project:ip-fabric:__default__ 09/01/2025 01:57:19.268 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', '__link_local__']: Unknown id: virtual_network default-domain:default-project:__link_local__ 09/01/2025 01:57:19.333 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', '__link_local__', '__link_local__']: Unknown id: routing_instance default-domain:default-project:__link_local__:__link_local__ 09/01/2025 01:57:19.360 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'dci-network']: Unknown id: virtual_network default-domain:default-project:dci-network 09/01/2025 01:57:19.441 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'dci-network', 'dci-network']: Unknown id: routing_instance default-domain:default-project:dci-network:dci-network 09/01/2025 01:57:19.468 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'dci-network', '__default__']: Unknown id: routing_instance default-domain:default-project:dci-network:__default__ 09/01/2025 01:57:19.490 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'system-defined-bottle-request-size-limit']: Unknown id: alarm default-global-system-config:system-defined-bottle-request-size-limit 09/01/2025 01:57:19.511 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network']: Unknown id: routing_instance default-domain:default-project:default-virtual-network:default-virtual-network 09/01/2025 01:57:19.539 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', '__no_rule__']: Unknown id: security_group default-domain:default-project:__no_rule__ 09/01/2025 01:57:19.561 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-discovery-service-assignment']: Unknown id: discovery_service_assignment default-discovery-service-assignment 09/01/2025 01:57:19.573 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-global-qos-config']: Unknown id: global_qos_config default-global-system-config:default-global-qos-config 09/01/2025 01:57:19.591 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'service-chain-flat-ipam']: Unknown id: network_ipam default-domain:default-project:service-chain-flat-ipam 09/01/2025 01:57:19.609 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['label']: Unknown id: tag_type label 09/01/2025 01:57:19.627 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['application']: Unknown id: tag_type application 09/01/2025 01:57:19.640 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['tier']: Unknown id: tag_type tier 09/01/2025 01:57:19.651 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['deployment']: Unknown id: tag_type deployment 09/01/2025 01:57:19.662 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['site']: Unknown id: tag_type site 09/01/2025 01:57:19.673 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['neutron_fwaas']: Unknown id: tag_type neutron_fwaas 09/01/2025 01:57:19.686 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['neutron_tag']: Unknown id: tag_type neutron_tag 09/01/2025 01:57:19.695 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Executing db_resync on api-server instance 10.0.0.56_0 09/01/2025 01:57:19.695 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Contrail Version is(nightly) 09/01/2025 01:57:19.719 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: zk_sync_node data (None) 09/01/2025 01:57:19.719 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DBERESYNC: running dbe-resync 09/01/2025 01:57:19.721 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Error in _create_default_rbac_rule: NoIdError: Unknown id: api_access_list default-global-system-config:default-api-access-list 09/01/2025 01:57:19.723 7f75df5eec80 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-api-access-list']: Unknown id: api_access_list default-global-system-config:default-api-access-list 09/01/2025 01:57:19.724 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type tag_type len 7 09/01/2025 01:57:19.724 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for tag_type 09/01/2025 01:57:19.728 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for tag_type 09/01/2025 01:57:19.728 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type global_qos_config len 1 09/01/2025 01:57:19.728 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for global_qos_config 09/01/2025 01:57:19.730 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for global_qos_config 09/01/2025 01:57:19.730 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type domain len 1 09/01/2025 01:57:19.730 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for domain 09/01/2025 01:57:19.844 7f75daa59540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.845 7f75daa59540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:19.845 7f75daa59540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.848 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Starting consumer greenlet 09/01/2025 01:57:19.849 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 01:57:19.849 7f75daa57860 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Starting publisher greenlet 09/01/2025 01:57:19.849 7f75daa57860 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Publisher connection 09/01/2025 01:57:19.850 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for domain 09/01/2025 01:57:19.850 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type routing_instance len 6 09/01/2025 01:57:19.850 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for routing_instance 09/01/2025 01:57:19.851 7f75df5eec80 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Added consumer: amqp_request.cn-jenkins-deploy-platform-ansible-os-3993-1..dummy 09/01/2025 01:57:19.851 7f75df5eec80 [contrail-api] [DEBUG]: Add uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 01:57:19.853 7f75daa59540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:19.856 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for routing_instance 09/01/2025 01:57:19.856 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type security_group len 1 09/01/2025 01:57:19.856 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for security_group 09/01/2025 01:57:19.858 7f75daa57860 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Publisher connection established 09/01/2025 01:57:19.860 7f75daa59540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:19.860 7f75daa59540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:19.860 7f75daa59540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:19.860 7f75daa59540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ConfigApiWorkerTrace: data = << name = worker-0 worker_id = 0 introspect_port = 8084 admin_port = 8095 introspect_url = http://10.0.0.56:8084 >> 09/01/2025 01:57:19.860 7f75daa59540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:19.860 7f75daa59540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:19.860 7f75daa59540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:19.860 7f75daa59540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:19.860 7f75daa59540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:19.861 7f75daa59540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:19.861 7f75daa59540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.861 7f75daa59540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:19.861 7f75daa59540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.861 7f75daa59540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.861 7f75daa59540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:19.861 7f75daa59540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.862 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for security_group 09/01/2025 01:57:19.862 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type alarm len 1 09/01/2025 01:57:19.863 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for alarm 09/01/2025 01:57:19.864 7f75daa57720 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Consumer connection established 09/01/2025 01:57:19.865 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for alarm 09/01/2025 01:57:19.866 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type discovery_service_assignment len 1 09/01/2025 01:57:19.866 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for discovery_service_assignment 09/01/2025 01:57:19.868 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for discovery_service_assignment 09/01/2025 01:57:19.869 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type virtual_network len 4 09/01/2025 01:57:19.869 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for virtual_network 09/01/2025 01:57:19.870 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Created consumers ['amqp_request.cn-jenkins-deploy-platform-ansible-os-3993-1..dummy'] 09/01/2025 01:57:19.883 7f75da7db5e0 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 09/01/2025 01:57:19.901 7f75da7dbd60 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 09/01/2025 01:57:19.902 7f75da7dbc20 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 09/01/2025 01:57:19.902 7f75da7dbae0 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 09/01/2025 01:57:19.902 7f75da7db9a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.902 7f75da7db9a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:default-virtual-network elements = { fq_name : ["default-domain", "default-project", "default-virtual-network"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10454952950717170781, "uuid_lslong": 10821947339228510081}, "enable": true, "created": "2025-09-01T01:57:18.995470", "last_modified": "2025-09-01T01:57:19.875082", "description": null, "user_visible": true, "creator": null}is_shared : falseparent_type : "project"parent_uuid : "a1121894-3f1d-413d-9f26-658ef40c9b7c"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "91177450-e2eb-445d-962f-47c494b2f781"virtual_network_network_id : 1 } deleted = False >> 09/01/2025 01:57:19.902 7f75da7db9a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.902 7f75da7db9a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.902 7f75da7db9a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:__link_local__ elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "__link_local__"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 8531175217902930706, "uuid_lslong": 12243259814534388473}, "enable": true, "created": "2025-09-01T01:57:19.283466", "last_modified": "2025-09-01T01:57:19.876282", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : falseis_shared : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"parent_uuid : "a1121894-3f1d-413d-9f26-658ef40c9b7c"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "7664d2a9-1bb8-4b12-a9e8-cbef0a231af9"virtual_network_category : "tenant"virtual_network_network_id : 3 } deleted = False >> 09/01/2025 01:57:19.903 7f75da7db9a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.903 7f75da7db9a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.903 7f75da7db9a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:dci-network elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "dci-network"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12476552366426768056, "uuid_lslong": 10220293011749065335}, "enable": true, "created": "2025-09-01T01:57:19.381403", "last_modified": "2025-09-01T01:57:19.876820", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : falseis_shared : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"parent_uuid : "a1121894-3f1d-413d-9f26-658ef40c9b7c"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "ad259e46-0e6d-42b8-8dd5-c65841b5d277"virtual_network_category : "tenant"virtual_network_network_id : 4 } deleted = False >> 09/01/2025 01:57:19.903 7f75da7db9a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.903 7f75da7db9a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.903 7f75da7db9a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:ip-fabric elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "ip-fabric"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 14095076780823300139, "uuid_lslong": 11232007001526243906}, "enable": true, "created": "2025-09-01T01:57:19.166835", "last_modified": "2025-09-01T01:57:19.877192", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : trueis_shared : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"parent_uuid : "a1121894-3f1d-413d-9f26-658ef40c9b7c"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "c39bc5a7-3ad4-482b-9be0-1adbb0ac1242"virtual_network_category : "tenant"virtual_network_network_id : 2 } deleted = False >> 09/01/2025 01:57:19.903 7f75da7db9a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.903 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for virtual_network 09/01/2025 01:57:19.903 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type project len 1 09/01/2025 01:57:19.903 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for project 09/01/2025 01:57:19.904 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 01:57:19.907 7f75daa57720 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Consumer connection established 09/01/2025 01:57:19.908 7f75dad1e180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'default-application-policy-set']: Unknown id: application_policy_set default-domain:default-project:default-application-policy-set 09/01/2025 01:57:19.912 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Created consumers ['amqp_request.cn-jenkins-deploy-platform-ansible-os-3993-1..dummy'] 09/01/2025 01:57:19.926 7f75da7db400 [contrail-api] [DEBUG]: Add uve in the [ObjectProjectTable:ContrailConfigTrace] map 09/01/2025 01:57:19.926 7f75da7dbae0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.926 7f75da7dbae0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { fq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11606366214680297789, "uuid_lslong": 11467965165799775100}, "enable": true, "created": "2025-09-01T01:57:18.966567", "last_modified": "2025-09-01T01:57:18.966567", "description": null, "user_visible": true, "creator": null}parent_type : "domain"parent_uuid : "29b069c3-005d-4827-93b3-22f4ba94fd44"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "a1121894-3f1d-413d-9f26-658ef40c9b7c" } deleted = False >> 09/01/2025 01:57:19.926 7f75da7dbae0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.926 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for project 09/01/2025 01:57:19.926 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type network_ipam len 2 09/01/2025 01:57:19.926 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for network_ipam 09/01/2025 01:57:19.930 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for network_ipam 09/01/2025 01:57:19.930 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type namespace len 1 09/01/2025 01:57:19.930 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for namespace 09/01/2025 01:57:19.931 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for namespace 09/01/2025 01:57:19.931 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type global_system_config len 1 09/01/2025 01:57:19.932 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for global_system_config 09/01/2025 01:57:19.933 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for global_system_config 09/01/2025 01:57:19.933 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type policy_management len 1 09/01/2025 01:57:19.933 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for policy_management 09/01/2025 01:57:19.935 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for policy_management 09/01/2025 01:57:19.935 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type fabric len 1 09/01/2025 01:57:19.935 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for fabric 09/01/2025 01:57:19.936 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for fabric 09/01/2025 01:57:19.936 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type global_vrouter_config len 1 09/01/2025 01:57:19.936 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for global_vrouter_config 09/01/2025 01:57:19.938 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for global_vrouter_config 09/01/2025 01:57:19.938 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type application_policy_set len 1 09/01/2025 01:57:19.938 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for application_policy_set 09/01/2025 01:57:19.940 7f75da7db220 [contrail-api] [DEBUG]: Add uve in the [ObjectApplicationPolicySetTable:ContrailConfigTrace] map 09/01/2025 01:57:19.940 7f75da7db900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.940 7f75da7db900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]global_vrouter_config_refs : [{"to": ["default-global-system-config", "default-global-vrouter-config"], "attr": null, "uuid": "08473562-4927-44da-a90c-47622528dc25"}]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9486536468119375630, "uuid_lslong": 9671471288597606353}, "enable": true, "created": "2025-09-01T01:57:19.115337", "last_modified": "2025-09-01T01:57:19.131801", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"parent_uuid : "125e91ee-f492-4106-b990-7eb68c7692a0"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "83a6f2a7-eb0b-4f0e-8637-f7e5343fefd1" } deleted = False >> 09/01/2025 01:57:19.941 7f75da7db900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.941 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for application_policy_set 09/01/2025 01:57:19.944 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DBERESYNC: Completed successfully 09/01/2025 01:57:19.946 7f75db801180 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Cassandra DB walk completed. 09/01/2025 01:57:19.956 7f75db801180 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Time elapsed in resyncing db: 0:00:00.261332 09/01/2025 01:57:19.957 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'obj_dict': {'autonomous_system': 64512, 'config_version': '1.0', 'enable_4byte_as': False, 'enable_security_policy_draft': False, 'fq_name': ['default-global-system-config'], 'ibgp_auto_mesh': True, 'id_perms': {'created': '2025-09-01T01:57:18.874446', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:18.874446', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10867045081573795072, 'uuid_mslong': 9146234606833845795}}, 'igmp_enable': False, 'mac_aging_time': 300, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500'}, 'oper': 'CREATE', 'request-id': 'req-85ee0d38-351d-4ac4-8b58-30669f0f871a', 'type': 'global_system_config', 'uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500'} 09/01/2025 01:57:19.958 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'obj_dict': {'enable_security_logging': True, 'encryption_mode': 'none', 'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'global_max_vmi_flows': 0, 'id_perms': {'created': '2025-09-01T01:57:18.915682', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:18.915682', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12181189578986150949, 'uuid_mslong': 596504171900257498}}, 'max_vm_flows_percent': 0, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '08473562-4927-44da-a90c-47622528dc25'}, 'oper': 'CREATE', 'request-id': 'req-015c55d7-78d7-406a-b06c-ae2a0e3748c2', 'type': 'global_vrouter_config', 'uuid': '08473562-4927-44da-a90c-47622528dc25'} 09/01/2025 01:57:19.959 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain'], 'obj_dict': {'fq_name': ['default-domain'], 'id_perms': {'created': '2025-09-01T01:57:18.943548', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:18.943548', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10642888778904304964, 'uuid_mslong': 3004017237701773351}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '29b069c3-005d-4827-93b3-22f4ba94fd44'}, 'oper': 'CREATE', 'request-id': 'req-d172ba6a-b076-4ff8-ab67-250ec56762af', 'type': 'domain', 'uuid': '29b069c3-005d-4827-93b3-22f4ba94fd44'} 09/01/2025 01:57:19.960 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project'], 'obj_dict': {'_enable_security_policy_draft': False, '_type': 'project', '_uuid': None, 'fq_name': ['default-domain', 'default-project'], 'id_perms': {'created': '2025-09-01T01:57:18.966567', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:18.966567', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11467965165799775100, 'uuid_mslong': 11606366214680297789}}, 'name': 'default-project', 'parent_type': 'domain', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a1121894-3f1d-413d-9f26-658ef40c9b7c'}, 'oper': 'CREATE', 'request-id': 'req-97287787-d9f4-4cb3-8e90-ae2f6215f439', 'type': 'project', 'uuid': 'a1121894-3f1d-413d-9f26-658ef40c9b7c'} 09/01/2025 01:57:19.961 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.961 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { _enable_security_policy_draft : false_type : "project"_uuid : nullfq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11606366214680297789, "uuid_lslong": 11467965165799775100}, "enable": true, "created": "2025-09-01T01:57:18.966567", "last_modified": "2025-09-01T01:57:18.966567", "description": null, "user_visible": true, "creator": null}name : "default-project"parent_type : "domain"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "a1121894-3f1d-413d-9f26-658ef40c9b7c" } deleted = False >> 09/01/2025 01:57:19.961 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.962 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network'], 'obj_dict': {'_fabric_snat': False, '_flood_unknown_unicast': False, '_igmp_enable': False, '_is_provider_network': False, '_layer2_control_word': False, '_mac_aging_time': 300, '_mac_ip_learning_enable': False, '_mac_learning_enabled': False, '_pbb_etree_enable': False, '_pbb_evpn_enable': False, '_port_security_enabled': True, '_type': 'virtual-network', '_uuid': None, '_virtual_network_category': 'tenant', 'fq_name': ['default-domain', 'default-project', 'default-virtual-network'], 'id_perms': {'created': '2025-09-01T01:57:18.995470', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:18.995470', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10821947339228510081, 'uuid_mslong': 10454952950717170781}}, 'name': 'default-virtual-network', 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '91177450-e2eb-445d-962f-47c494b2f781', 'virtual_network_network_id': 1}, 'oper': 'CREATE', 'request-id': 'req-8082c4af-47f6-4731-83c3-3002019955f2', 'type': 'virtual_network', 'uuid': '91177450-e2eb-445d-962f-47c494b2f781'} 09/01/2025 01:57:19.963 7f75daa595e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.963 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:default-virtual-network elements = { _fabric_snat : false_flood_unknown_unicast : false_igmp_enable : false_is_provider_network : false_layer2_control_word : false_mac_aging_time : 300_mac_ip_learning_enable : false_mac_learning_enabled : false_pbb_etree_enable : false_pbb_evpn_enable : false_port_security_enabled : true_type : "virtual-network"_uuid : null_virtual_network_category : "tenant"fq_name : ["default-domain", "default-project", "default-virtual-network"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10454952950717170781, "uuid_lslong": 10821947339228510081}, "enable": true, "created": "2025-09-01T01:57:18.995470", "last_modified": "2025-09-01T01:57:18.995470", "description": null, "user_visible": true, "creator": null}name : "default-virtual-network"parent_type : "project"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "91177450-e2eb-445d-962f-47c494b2f781"virtual_network_network_id : 1 } deleted = False >> 09/01/2025 01:57:19.963 7f75daa595e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.965 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'obj_dict': {'_type': 'network-ipam', '_uuid': None, 'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'id_perms': {'created': '2025-09-01T01:57:19.020987', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.020987', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9745809469775807592, 'uuid_mslong': 16265253945370690267}}, 'name': 'default-network-ipam', 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e1b9ca49-4b57-42db-8740-1213c6a00468'}, 'oper': 'CREATE', 'request-id': 'req-663d2e5a-422a-48ef-8883-1ebde95bcbcf', 'type': 'network_ipam', 'uuid': 'e1b9ca49-4b57-42db-8740-1213c6a00468'} 09/01/2025 01:57:19.966 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-namespace'], 'obj_dict': {'_type': 'namespace', '_uuid': None, 'fq_name': ['default-domain', 'default-namespace'], 'id_perms': {'created': '2025-09-01T01:57:19.039932', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.039932', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9964883438110359958, 'uuid_mslong': 6198738153118450004}}, 'name': 'default-namespace', 'parent_type': 'domain', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '56065782-0849-4154-8a4a-60b0d4bdc196'}, 'oper': 'CREATE', 'request-id': 'req-9eadbea0-d8aa-4cd0-b859-df72180d98f4', 'type': 'namespace', 'uuid': '56065782-0849-4154-8a4a-60b0d4bdc196'} 09/01/2025 01:57:19.967 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-fabric'], 'obj_dict': {'disable_vlan_vn_uniqueness_check': False, 'fabric_enterprise_style': True, 'fq_name': ['default-global-system-config', 'default-fabric'], 'id_perms': {'created': '2025-09-01T01:57:19.067952', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.067952', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13125710443338707915, 'uuid_mslong': 1807343850622369981}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '1914f9a8-0b55-40bd-b627-e41aefd6b3cb'}, 'oper': 'CREATE', 'request-id': 'req-06b9712f-8c73-467b-b604-c1db28066f61', 'type': 'fabric', 'uuid': '1914f9a8-0b55-40bd-b627-e41aefd6b3cb'} 09/01/2025 01:57:19.967 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management'], 'obj_dict': {'fq_name': ['default-policy-management'], 'id_perms': {'created': '2025-09-01T01:57:19.092836', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.092836', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13371326616168731296, 'uuid_mslong': 1323655795984974086}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '125e91ee-f492-4106-b990-7eb68c7692a0'}, 'oper': 'CREATE', 'request-id': 'req-6a677c01-809f-4fa1-a924-e47f2bcd3bc5', 'type': 'policy_management', 'uuid': '125e91ee-f492-4106-b990-7eb68c7692a0'} 09/01/2025 01:57:19.968 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'obj_dict': {'all_applications': True, 'fq_name': ['default-policy-management', 'default-application-policy-set'], 'id_perms': {'created': '2025-09-01T01:57:19.115337', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.115337', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9671471288597606353, 'uuid_mslong': 9486536468119375630}}, 'parent_type': 'policy-management', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '83a6f2a7-eb0b-4f0e-8637-f7e5343fefd1'}, 'oper': 'CREATE', 'request-id': 'req-f900d811-f7de-43c0-8154-67559e743818', 'type': 'application_policy_set', 'uuid': '83a6f2a7-eb0b-4f0e-8637-f7e5343fefd1'} 09/01/2025 01:57:19.969 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'oper': 'UPDATE', 'request-id': 'req-8cc1c8e1-b781-4d9d-9ab7-dda13424987a', 'type': 'application_policy_set', 'uuid': '83a6f2a7-eb0b-4f0e-8637-f7e5343fefd1'} 09/01/2025 01:57:19.969 7f75daa595e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.970 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9486536468119375630, "uuid_lslong": 9671471288597606353}, "enable": true, "created": "2025-09-01T01:57:19.115337", "last_modified": "2025-09-01T01:57:19.115337", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "83a6f2a7-eb0b-4f0e-8637-f7e5343fefd1" } deleted = False >> 09/01/2025 01:57:19.970 7f75daa595e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.972 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric'], 'obj_dict': {'fabric_snat': False, 'flood_unknown_unicast': False, 'fq_name': ['default-domain', 'default-project', 'ip-fabric'], 'id_perms': {'created': '2025-09-01T01:57:19.166835', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.166835', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11232007001526243906, 'uuid_mslong': 14095076780823300139}}, 'igmp_enable': False, 'is_provider_network': True, 'layer2_control_word': False, 'mac_aging_time': 300, 'mac_ip_learning_enable': False, 'mac_learning_enabled': False, 'parent_type': 'project', 'pbb_etree_enable': False, 'pbb_evpn_enable': False, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'uuid': 'c39bc5a7-3ad4-482b-9be0-1adbb0ac1242', 'virtual_network_category': 'tenant', 'virtual_network_network_id': 2}, 'oper': 'CREATE', 'request-id': 'req-61009e7d-120f-4963-94a7-8ae4866ed8ae', 'type': 'virtual_network', 'uuid': 'c39bc5a7-3ad4-482b-9be0-1adbb0ac1242'} 09/01/2025 01:57:19.973 7f75daa595e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.973 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]global_vrouter_config_refs : [{"to": ["default-global-system-config", "default-global-vrouter-config"], "attr": null, "uuid": "08473562-4927-44da-a90c-47622528dc25"}]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9486536468119375630, "uuid_lslong": 9671471288597606353}, "enable": true, "created": "2025-09-01T01:57:19.115337", "last_modified": "2025-09-01T01:57:19.131801", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"parent_uuid : "125e91ee-f492-4106-b990-7eb68c7692a0"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "83a6f2a7-eb0b-4f0e-8637-f7e5343fefd1" } deleted = False >> 09/01/2025 01:57:19.973 7f75daa595e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.973 7f75daa595e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.973 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:ip-fabric elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "ip-fabric"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 14095076780823300139, "uuid_lslong": 11232007001526243906}, "enable": true, "created": "2025-09-01T01:57:19.166835", "last_modified": "2025-09-01T01:57:19.166835", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : truelayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "c39bc5a7-3ad4-482b-9be0-1adbb0ac1242"virtual_network_category : "tenant"virtual_network_network_id : 2 } deleted = False >> 09/01/2025 01:57:19.973 7f75daa595e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.975 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'id_perms': {'created': '2025-09-01T01:57:19.239382', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.239382', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9621015281281886679, 'uuid_mslong': 2529689819727677828}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '231b437c-f8ca-4984-8584-b66c8b6969d7'}, 'oper': 'CREATE', 'request-id': 'req-ee686793-8f8a-484c-9b01-38e5d75476a5', 'type': 'routing_instance', 'uuid': '231b437c-f8ca-4984-8584-b66c8b6969d7'} 09/01/2025 01:57:19.976 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__'], 'id_perms': {'created': '2025-09-01T01:57:19.260272', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.260272', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12080804100940942551, 'uuid_mslong': 9296926739905923499}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': False, 'uuid': '8105519c-daec-49ab-a7a7-a352adaaccd7'}, 'oper': 'CREATE', 'request-id': 'req-b8799f29-f660-4fdd-8fb8-43f84e66e94d', 'type': 'routing_instance', 'uuid': '8105519c-daec-49ab-a7a7-a352adaaccd7'} 09/01/2025 01:57:19.977 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__link_local__'], 'obj_dict': {'fabric_snat': False, 'flood_unknown_unicast': False, 'fq_name': ['default-domain', 'default-project', '__link_local__'], 'id_perms': {'created': '2025-09-01T01:57:19.283466', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.283466', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12243259814534388473, 'uuid_mslong': 8531175217902930706}}, 'igmp_enable': False, 'is_provider_network': False, 'layer2_control_word': False, 'mac_aging_time': 300, 'mac_ip_learning_enable': False, 'mac_learning_enabled': False, 'parent_type': 'project', 'pbb_etree_enable': False, 'pbb_evpn_enable': False, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'uuid': '7664d2a9-1bb8-4b12-a9e8-cbef0a231af9', 'virtual_network_category': 'tenant', 'virtual_network_network_id': 3}, 'oper': 'CREATE', 'request-id': 'req-a3551969-82d0-4cbe-9829-01f2c8d543cc', 'type': 'virtual_network', 'uuid': '7664d2a9-1bb8-4b12-a9e8-cbef0a231af9'} 09/01/2025 01:57:19.977 7f75daa595e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.977 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:__link_local__ elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "__link_local__"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 8531175217902930706, "uuid_lslong": 12243259814534388473}, "enable": true, "created": "2025-09-01T01:57:19.283466", "last_modified": "2025-09-01T01:57:19.283466", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "7664d2a9-1bb8-4b12-a9e8-cbef0a231af9"virtual_network_category : "tenant"virtual_network_network_id : 3 } deleted = False >> 09/01/2025 01:57:19.977 7f75daa595e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.979 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__link_local__', '__link_local__'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', '__link_local__', '__link_local__'], 'id_perms': {'created': '2025-09-01T01:57:19.351837', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.351837', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9723707038546547581, 'uuid_mslong': 3641056327929446881}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '3287a170-696c-41e1-86f1-8c087c88537d'}, 'oper': 'CREATE', 'request-id': 'req-12917a87-8fc7-453a-a129-acb7a312f753', 'type': 'routing_instance', 'uuid': '3287a170-696c-41e1-86f1-8c087c88537d'} 09/01/2025 01:57:19.980 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network'], 'obj_dict': {'fabric_snat': False, 'flood_unknown_unicast': False, 'fq_name': ['default-domain', 'default-project', 'dci-network'], 'id_perms': {'created': '2025-09-01T01:57:19.381403', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.381403', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10220293011749065335, 'uuid_mslong': 12476552366426768056}}, 'igmp_enable': False, 'is_provider_network': False, 'layer2_control_word': False, 'mac_aging_time': 300, 'mac_ip_learning_enable': False, 'mac_learning_enabled': False, 'parent_type': 'project', 'pbb_etree_enable': False, 'pbb_evpn_enable': False, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'uuid': 'ad259e46-0e6d-42b8-8dd5-c65841b5d277', 'virtual_network_category': 'tenant', 'virtual_network_network_id': 4}, 'oper': 'CREATE', 'request-id': 'req-44aa4388-7617-44c6-9f5c-685afaf538ad', 'type': 'virtual_network', 'uuid': 'ad259e46-0e6d-42b8-8dd5-c65841b5d277'} 09/01/2025 01:57:19.980 7f75daa595e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.980 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:dci-network elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "dci-network"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12476552366426768056, "uuid_lslong": 10220293011749065335}, "enable": true, "created": "2025-09-01T01:57:19.381403", "last_modified": "2025-09-01T01:57:19.381403", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "ad259e46-0e6d-42b8-8dd5-c65841b5d277"virtual_network_category : "tenant"virtual_network_network_id : 4 } deleted = False >> 09/01/2025 01:57:19.980 7f75daa595e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.982 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'id_perms': {'created': '2025-09-01T01:57:19.457584', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.457584', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13040679847623360549, 'uuid_mslong': 10122422825586803134}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '8c7a11ed-d168-41be-b4f9-cd3a9b2f8025'}, 'oper': 'CREATE', 'request-id': 'req-4bdaf9bc-9eb4-47df-8fb4-6eb409b84b78', 'type': 'routing_instance', 'uuid': '8c7a11ed-d168-41be-b4f9-cd3a9b2f8025'} 09/01/2025 01:57:19.983 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'id_perms': {'created': '2025-09-01T01:57:19.485571', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.485571', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12776497886638577655, 'uuid_mslong': 16517360699546354692}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': False, 'uuid': 'e539740b-2375-4404-b14f-3d22969f17f7'}, 'oper': 'CREATE', 'request-id': 'req-6848a5b1-d6d2-4590-81d9-79655a226091', 'type': 'routing_instance', 'uuid': 'e539740b-2375-4404-b14f-3d22969f17f7'} 09/01/2025 01:57:19.984 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bottle-request-size-limit'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'UveConfigReq.err_info.*.', 'operand2': {'json_value': 'True'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'fq_name': ['default-global-system-config', 'system-defined-bottle-request-size-limit'], 'id_perms': {'created': '2025-09-01T01:57:19.500783', 'description': 'Bottle request size limit exceeded.', 'last_modified': '2025-09-01T01:57:19.500783', 'uuid': {'uuid_lslong': 12248324434343451382, 'uuid_mslong': 12864092550890866045}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b2867004-a599-457d-a9fa-ca2dbac75af6', 'uve_keys': {'uve_key': ['config-node']}}, 'oper': 'CREATE', 'request-id': 'req-1812dfe7-1c61-4d75-bb98-73f542b359a1', 'type': 'alarm', 'uuid': 'b2867004-a599-457d-a9fa-ca2dbac75af6'} 09/01/2025 01:57:19.984 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'id_perms': {'created': '2025-09-01T01:57:19.534084', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.534084', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13238994389195681154, 'uuid_mslong': 14202240796981809066}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': 'c5187ec2-16a9-4faa-b7ba-5b40f765ed82'}, 'oper': 'CREATE', 'request-id': 'req-9db988b1-d8de-45c7-b461-43828daa1929', 'type': 'routing_instance', 'uuid': 'c5187ec2-16a9-4faa-b7ba-5b40f765ed82'} 09/01/2025 01:57:19.985 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__'], 'obj_dict': {'configured_security_group_id': 0, 'fq_name': ['default-domain', 'default-project', '__no_rule__'], 'id_perms': {'created': '2025-09-01T01:57:19.553823', 'creator': None, 'description': 'Security group with no rules', 'enable': True, 'last_modified': '2025-09-01T01:57:19.553823', 'permissions': None, 'user_visible': True, 'uuid': {'uuid_lslong': 10451439962015470572, 'uuid_mslong': 7065600600121034336}}, 'parent_type': 'project', 'perms2': {'global_access': 5, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'security_group_entries': {'policy_rule': []}, 'security_group_id': 8000001, 'uuid': '620e0e4f-8700-4660-910a-f945774113ec'}, 'oper': 'CREATE', 'request-id': 'req-bc08144b-1d04-4179-bc2e-339ff457f1f2', 'type': 'security_group', 'uuid': '620e0e4f-8700-4660-910a-f945774113ec'} 09/01/2025 01:57:19.987 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-discovery-service-assignment'], 'obj_dict': {'fq_name': ['default-discovery-service-assignment'], 'id_perms': {'created': '2025-09-01T01:57:19.568716', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.568716', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13207971436539284858, 'uuid_mslong': 15905819078235931959}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'dcbcd22d-d3e2-4537-b74c-240aee6e0d7a'}, 'oper': 'CREATE', 'request-id': 'req-65210073-fad2-4e06-b19c-360eeba39bb3', 'type': 'discovery_service_assignment', 'uuid': 'dcbcd22d-d3e2-4537-b74c-240aee6e0d7a'} 09/01/2025 01:57:19.988 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-qos-config'], 'obj_dict': {'fq_name': ['default-global-system-config', 'default-global-qos-config'], 'id_perms': {'created': '2025-09-01T01:57:19.584826', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.584826', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11183850458333252522, 'uuid_mslong': 11384797924189358600}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '9dfeed64-8ee9-4a08-9b35-04bcbe0b0baa'}, 'oper': 'CREATE', 'request-id': 'req-bb4dd9d5-e4cc-40e1-bdc5-5bf9f3ed8240', 'type': 'global_qos_config', 'uuid': '9dfeed64-8ee9-4a08-9b35-04bcbe0b0baa'} 09/01/2025 01:57:19.989 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'service-chain-flat-ipam'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'service-chain-flat-ipam'], 'id_perms': {'created': '2025-09-01T01:57:19.602345', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.602345', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9328314797137868181, 'uuid_mslong': 15533010930695949457}}, 'ipam_subnet_method': 'flat-subnet', 'ipam_subnets': {'subnets': [{'addr_from_start': None, 'alloc_unit': 1, 'allocation_pools': [], 'created': None, 'default_gateway': None, 'dhcp_option_list': None, 'dhcp_relay_server': [], 'dns_nameservers': [], 'dns_server_address': None, 'enable_dhcp': True, 'host_routes': None, 'last_modified': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 8}, 'subnet_name': None, 'subnet_uuid': None, 'subscriber_tag': None, 'vlan_tag': None}, {'addr_from_start': None, 'alloc_unit': 1, 'allocation_pools': [], 'created': None, 'default_gateway': None, 'dhcp_option_list': None, 'dhcp_relay_server': [], 'dns_nameservers': [], 'dns_server_address': None, 'enable_dhcp': True, 'host_routes': None, 'last_modified': None, 'subnet': {'ip_prefix': '::ffff', 'ip_prefix_len': 104}, 'subnet_name': None, 'subnet_uuid': None, 'subscriber_tag': None, 'vlan_tag': None}]}, 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd7905724-e064-4c91-8174-d4e26c6c3595'}, 'oper': 'CREATE', 'request-id': 'req-ad847e15-603f-4c0f-aaf7-4f9b5203df33', 'type': 'network_ipam', 'uuid': 'd7905724-e064-4c91-8174-d4e26c6c3595'} 09/01/2025 01:57:19.992 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['label'], 'obj_dict': {'display_name': 'label', 'fq_name': ['label'], 'id_perms': {'created': '2025-09-01T01:57:19.621582', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.621582', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 9684506176955880422, 'uuid_mslong': 11639477088750881084}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0000', 'uuid': 'a187babe-5e4b-493c-8666-470ea68813e6'}, 'oper': 'CREATE', 'request-id': 'req-82d55ab1-6b06-4289-86e6-a694759a6de7', 'type': 'tag_type', 'uuid': 'a187babe-5e4b-493c-8666-470ea68813e6'} 09/01/2025 01:57:19.994 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['application'], 'obj_dict': {'display_name': 'application', 'fq_name': ['application'], 'id_perms': {'created': '2025-09-01T01:57:19.635010', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.635010', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 13187130878668002583, 'uuid_mslong': 17025456246190460525}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0001', 'uuid': 'ec469240-4d23-4e6d-b702-19aa6d975517'}, 'oper': 'CREATE', 'request-id': 'req-485410d3-55a8-44f0-a6e1-7036934dc1bd', 'type': 'tag_type', 'uuid': 'ec469240-4d23-4e6d-b702-19aa6d975517'} 09/01/2025 01:57:19.995 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['tier'], 'obj_dict': {'display_name': 'tier', 'fq_name': ['tier'], 'id_perms': {'created': '2025-09-01T01:57:19.647013', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.647013', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 9225052825710834370, 'uuid_mslong': 4334600452385885078}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0002', 'uuid': '3c279825-4fb5-4b96-8005-f8ab26b532c2'}, 'oper': 'CREATE', 'request-id': 'req-7f6b09c1-9ece-4418-a64d-8f4a491d9491', 'type': 'tag_type', 'uuid': '3c279825-4fb5-4b96-8005-f8ab26b532c2'} 09/01/2025 01:57:19.997 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['deployment'], 'obj_dict': {'display_name': 'deployment', 'fq_name': ['deployment'], 'id_perms': {'created': '2025-09-01T01:57:19.657281', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.657281', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 12185577634141415688, 'uuid_mslong': 8052265578173321202}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0003', 'uuid': '6fbf64e1-7020-4ff2-a91b-de4bd9f1a908'}, 'oper': 'CREATE', 'request-id': 'req-0193331a-7328-4e5c-9475-9cf952111b76', 'type': 'tag_type', 'uuid': '6fbf64e1-7020-4ff2-a91b-de4bd9f1a908'} 09/01/2025 01:57:19.998 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['site'], 'obj_dict': {'display_name': 'site', 'fq_name': ['site'], 'id_perms': {'created': '2025-09-01T01:57:19.668413', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.668413', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 13138019419830804121, 'uuid_mslong': 10498085874017192065}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0004', 'uuid': '91b0b17b-2e57-4881-b653-9f0ddb32de99'}, 'oper': 'CREATE', 'request-id': 'req-29ecd828-fb9a-477f-8cfd-40ef7ff900b0', 'type': 'tag_type', 'uuid': '91b0b17b-2e57-4881-b653-9f0ddb32de99'} 09/01/2025 01:57:20.000 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['neutron_fwaas'], 'obj_dict': {'display_name': 'neutron_fwaas', 'fq_name': ['neutron_fwaas'], 'id_perms': {'created': '2025-09-01T01:57:19.680164', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.680164', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 12722226411090425079, 'uuid_mslong': 12553196925081765101}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0005', 'uuid': 'ae35ea17-fb06-44ed-b08e-6d84012334f7'}, 'oper': 'CREATE', 'request-id': 'req-5fccc3ae-420e-4d65-bcc4-2cfbed7a40e3', 'type': 'tag_type', 'uuid': 'ae35ea17-fb06-44ed-b08e-6d84012334f7'} 09/01/2025 01:57:20.002 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['neutron_tag'], 'obj_dict': {'display_name': 'neutron_tag', 'fq_name': ['neutron_tag'], 'id_perms': {'created': '2025-09-01T01:57:19.692005', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.692005', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 12917025899577138027, 'uuid_mslong': 16654701351454984872}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0006', 'uuid': 'e72162a3-85b1-4aa8-b342-7e9e4ad0db6b'}, 'oper': 'CREATE', 'request-id': 'req-101d1fad-de96-4b1a-9896-ba72a0f876db', 'type': 'tag_type', 'uuid': 'e72162a3-85b1-4aa8-b342-7e9e4ad0db6b'} 09/01/2025 01:57:20.003 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'oper': 'UPDATE', 'request-id': 'req-be15cd48-8fff-497b-9500-2cd2aa017804', 'type': 'network_ipam', 'uuid': 'e1b9ca49-4b57-42db-8740-1213c6a00468'} 09/01/2025 01:57:20.007 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-api-access-list'], 'obj_dict': {'api_access_list_entries': {'rbac_rule': [{'rule_field': '', 'rule_object': 'fqname-to-id', 'rule_perms': [{'role_crud': 'CRUD', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'id-to-fqname', 'rule_perms': [{'role_crud': 'CRUD', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'useragent-kv', 'rule_perms': [{'role_crud': 'CRUD', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'documentation', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': '/', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'watch', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}, {'rule_field': 'logical_router_gateway_external', 'rule_object': 'logical-router', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}]}, 'fq_name': ['default-global-system-config', 'default-api-access-list'], 'id_perms': {'created': '2025-09-01T01:57:19.734229', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.734229', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10313762822725109713, 'uuid_mslong': 9601466828499600373}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '853f4335-b36a-43f5-8f21-d8a4822907d1'}, 'oper': 'CREATE', 'request-id': 'req-fc5458f6-7566-4de9-b8e2-32efbd3dc00b', 'type': 'api_access_list', 'uuid': '853f4335-b36a-43f5-8f21-d8a4822907d1'} 09/01/2025 01:57:20.008 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-application-policy-set'], 'obj_dict': {'all_applications': True, 'fq_name': ['default-domain', 'default-project', 'default-application-policy-set'], 'id_perms': {'created': '2025-09-01T01:57:19.922585', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:19.922585', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10506699027801958067, 'uuid_mslong': 7648986211723005738}}, 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6a26a86e-f99c-4f2a-91cf-4b195213e6b3'}, 'oper': 'CREATE', 'request-id': 'req-50b08f9e-ad33-4843-aba7-f95230091987', 'type': 'application_policy_set', 'uuid': '6a26a86e-f99c-4f2a-91cf-4b195213e6b3'} 09/01/2025 01:57:20.008 7f75daccbc20 [contrail-api] [DEBUG]: Add uve in the [ObjectApplicationPolicySetTable:ContrailConfigTrace] map 09/01/2025 01:57:20.009 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project'], 'oper': 'UPDATE', 'request-id': 'req-bba4423f-fdf5-4231-b901-a01c0fa87be0', 'type': 'project', 'uuid': 'a1121894-3f1d-413d-9f26-658ef40c9b7c'} 09/01/2025 01:57:20.009 7f75daa595e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:20.015 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:default-application-policy-set elements = { all_applications : truefq_name : ["default-domain", "default-project", "default-application-policy-set"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7648986211723005738, "uuid_lslong": 10506699027801958067}, "enable": true, "created": "2025-09-01T01:57:19.922585", "last_modified": "2025-09-01T01:57:19.922585", "description": null, "user_visible": true, "creator": null}parent_type : "project"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "6a26a86e-f99c-4f2a-91cf-4b195213e6b3" } deleted = False >> 09/01/2025 01:57:20.015 7f75daa595e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:20.017 7f75daa595e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:20.017 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { fq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11606366214680297789, "uuid_lslong": 11467965165799775100}, "enable": true, "created": "2025-09-01T01:57:18.966567", "last_modified": "2025-09-01T01:57:19.953692", "description": null, "user_visible": true, "creator": null}parent_type : "domain"parent_uuid : "29b069c3-005d-4827-93b3-22f4ba94fd44"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}quota : {"defaults": -1}uuid : "a1121894-3f1d-413d-9f26-658ef40c9b7c" } deleted = False >> 09/01/2025 01:57:20.017 7f75daa595e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:20.913 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:21.915 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:22.438 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:22.439 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:22.439 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:22.439 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:22.440 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:22.440 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:22.440 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:22.440 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:22.441 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:22.441 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:22.441 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:22.442 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:22.442 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:22.442 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:22.442 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:22.442 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:22.442 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:22.442 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:22.442 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:22.443 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:22.443 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:22.443 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:22.443 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:22.443 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:22.443 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:22.443 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:22.444 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:22.444 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:22.445 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:22.445 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:22.445 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:22.445 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:22.445 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:57:22.453 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:22.454 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:22.553 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'] not found in DB: Unknown id: config_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1. 09/01/2025 01:57:22.563 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = config_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1. url = http://10.0.0.56:8082/config-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_config_node.py remote_ip = 10.0.0.56 body = {"config-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1."], "parent_type": "global-system-config", "config_node_ip_address": "10.0.0.56", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3993-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:22.577 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'], 'obj_dict': {'config_node_ip_address': '10.0.0.56', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3993-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'], 'id_perms': {'created': '2025-09-01T01:57:22.571752', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:22.571752', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13037776234279267945, 'uuid_mslong': 1417693221577704725}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '13aca879-489e-4515-b4ef-7c68713eee69'}, 'oper': 'CREATE', 'request-id': 'req-4c7bdee5-06c4-4e1d-b255-0ec86cf9b6c0', 'type': 'config_node', 'uuid': '13aca879-489e-4515-b4ef-7c68713eee69'} 09/01/2025 01:57:22.578 7f75daccbc20 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:ContrailConfigTrace] map 09/01/2025 01:57:22.579 7f75da7dbd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:22.579 7f75da7dbd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. elements = { config_node_ip_address : "10.0.0.56"display_name : "cn-jenkins-deploy-platform-ansible-os-3993-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1417693221577704725, "uuid_lslong": 13037776234279267945}, "enable": true, "created": "2025-09-01T01:57:22.571752", "last_modified": "2025-09-01T01:57:22.571752", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "13aca879-489e-4515-b4ef-7c68713eee69" } deleted = False >> 09/01/2025 01:57:22.580 7f75da7dbd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:22.917 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:23.401 7f75da7dbcc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:23.401 7f75da7dbcc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:23.401 7f75da7dbcc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:23.860 7f75daa57e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:23.868 7f75daa57e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:23.868 7f75daa57e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:23.868 7f75daa57e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:23.868 7f75daa57e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:23.868 7f75daa57e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:23.868 7f75daa57e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:23.869 7f75daa57e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:23.869 7f75daa57e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:23.869 7f75daa57e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:23.869 7f75daa57e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:23.869 7f75daa57e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:23.869 7f75daa57e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:23.869 7f75daa57e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:23.918 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:24.144 7f75daa595e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_DNS_refs in resource network-ipam: 'virtual_DNS_refs' 09/01/2025 01:57:24.144 7f75daa595e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource network-ipam: 'tag_refs' 09/01/2025 01:57:24.144 7f75daa595e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_back_refs in resource network-ipam: 'virtual_network_back_refs' 09/01/2025 01:57:24.145 7f75daa595e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_router_back_refs in resource network-ipam: 'virtual_router_back_refs' 09/01/2025 01:57:24.145 7f75daa595e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource network-ipam: 'instance_ip_back_refs' 09/01/2025 01:57:24.146 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:24.146 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:24.147 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:24.147 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:24.150 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:24.152 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:24.152 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:24.152 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:24.153 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:24.153 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:24.159 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:24.159 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:24.159 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:24.159 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:24.160 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:24.163 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:24.163 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:24.163 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:24.163 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:24.164 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:24.164 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:24.164 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:24.164 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:24.164 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:24.165 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:24.165 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:24.165 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:24.168 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:24.174 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:24.175 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:24.175 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:24.175 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:24.176 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:57:24.176 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:24.177 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:24.195 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:24.195 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:24.196 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:24.196 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:24.196 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:24.196 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:24.197 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:24.198 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:24.198 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:24.198 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:24.198 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:24.199 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:24.202 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:24.202 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:24.202 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:24.202 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:24.207 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:24.207 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:24.208 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:24.208 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:24.209 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:24.209 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:24.209 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:24.209 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:24.210 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:24.210 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:24.210 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:24.210 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:24.210 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:24.210 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:24.213 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:24.213 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:24.213 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:57:24.214 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:24.214 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:24.288 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'opencontrail'] not found in DB: Unknown id: service_appliance_set default-global-system-config:opencontrail 09/01/2025 01:57:24.290 7f75da7db900 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_appliance_set identifier_name = default-global-system-config:opencontrail url = http://10.0.0.56:8082/service-appliance-sets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.56 body = {"service-appliance-set": {"uuid": null, "fq_name": ["default-global-system-config", "opencontrail"], "parent_type": "global-system-config", "service_appliance_driver": "svc_monitor.services.loadbalancer.drivers.ha_proxy.driver.OpencontrailLoadbalancerDriver", "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 5, "share": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "display_name": "opencontrail"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:24.296 7f75da7db220 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'] not found in DB: Unknown id: config_database_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1. 09/01/2025 01:57:24.317 7f75da7db220 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = config_database_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1. url = http://10.0.0.56:8082/config-database-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_config_database_node.py remote_ip = 10.0.0.56 body = {"config-database-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1."], "parent_type": "global-system-config", "config_database_node_ip_address": "10.0.0.56", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3993-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:24.333 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'opencontrail'], 'obj_dict': {'display_name': 'opencontrail', 'fq_name': ['default-global-system-config', 'opencontrail'], 'id_perms': {'created': '2025-09-01T01:57:24.324542', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:24.324542', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10024902600845278318, 'uuid_mslong': 15513935732498645698}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 5, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service_appliance_driver': 'svc_monitor.services.loadbalancer.drivers.ha_proxy.driver.OpencontrailLoadbalancerDriver', 'uuid': 'd74c925a-33d2-42c2-8b1f-9bcd1f45bc6e'}, 'oper': 'CREATE', 'request-id': 'req-ab86164b-2d60-465f-ada9-12057121f1cd', 'type': 'service_appliance_set', 'uuid': 'd74c925a-33d2-42c2-8b1f-9bcd1f45bc6e'} 09/01/2025 01:57:24.342 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'], 'obj_dict': {'config_database_node_ip_address': '10.0.0.56', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3993-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'], 'id_perms': {'created': '2025-09-01T01:57:24.332811', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:24.332811', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11274207856087790539, 'uuid_mslong': 14139981547036036428}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c43b4e4c-d9a9-4d4c-9c76-084fd207f3cb'}, 'oper': 'CREATE', 'request-id': 'req-d20df887-da24-4d38-91e6-b8a2dd180eff', 'type': 'config_database_node', 'uuid': 'c43b4e4c-d9a9-4d4c-9c76-084fd207f3cb'} 09/01/2025 01:57:24.343 7f75daccbc20 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:57:24.344 7f75da7dbae0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:24.345 7f75da7dbae0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. elements = { config_database_node_ip_address : "10.0.0.56"display_name : "cn-jenkins-deploy-platform-ansible-os-3993-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 14139981547036036428, "uuid_lslong": 11274207856087790539}, "enable": true, "created": "2025-09-01T01:57:24.332811", "last_modified": "2025-09-01T01:57:24.332811", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "c43b4e4c-d9a9-4d4c-9c76-084fd207f3cb" } deleted = False >> 09/01/2025 01:57:24.346 7f75da7dbae0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:24.491 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:24.492 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:24.492 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:24.493 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:24.493 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:24.493 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:24.493 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:24.494 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:24.494 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:24.494 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:24.494 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:24.494 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:24.495 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:24.495 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:24.495 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:24.495 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:24.496 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:24.496 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:24.496 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:24.496 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:24.497 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:24.497 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:24.497 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:24.498 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:24.499 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:24.499 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:24.499 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:24.499 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:24.500 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:24.500 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:24.501 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:24.501 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:24.502 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:57:24.502 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:24.503 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:24.562 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'native'] not found in DB: Unknown id: service_appliance_set default-global-system-config:native 09/01/2025 01:57:24.563 7f75da7db900 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_appliance_set identifier_name = default-global-system-config:native url = http://10.0.0.56:8082/service-appliance-sets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.56 body = {"service-appliance-set": {"uuid": null, "fq_name": ["default-global-system-config", "native"], "parent_type": "global-system-config", "service_appliance_driver": "svc_monitor.services.loadbalancer.drivers.native.driver.OpencontrailLoadbalancerDriver", "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 5, "share": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "display_name": "native"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:24.591 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'native'], 'obj_dict': {'display_name': 'native', 'fq_name': ['default-global-system-config', 'native'], 'id_perms': {'created': '2025-09-01T01:57:24.577603', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:24.577603', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10255140895785741537, 'uuid_mslong': 17579200508297038660}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 5, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service_appliance_driver': 'svc_monitor.services.loadbalancer.drivers.native.driver.OpencontrailLoadbalancerDriver', 'uuid': 'f3f5ddb9-4469-4b44-8e51-944f86e3ece1'}, 'oper': 'CREATE', 'request-id': 'req-91b6802d-05e9-4ee5-96ad-fc7283156b84', 'type': 'service_appliance_set', 'uuid': 'f3f5ddb9-4469-4b44-8e51-944f86e3ece1'} 09/01/2025 01:57:24.700 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_DNS_refs in resource network-ipam: 'virtual_DNS_refs' 09/01/2025 01:57:24.700 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource network-ipam: 'tag_refs' 09/01/2025 01:57:24.700 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_back_refs in resource network-ipam: 'virtual_network_back_refs' 09/01/2025 01:57:24.701 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_router_back_refs in resource network-ipam: 'virtual_router_back_refs' 09/01/2025 01:57:24.701 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource network-ipam: 'instance_ip_back_refs' 09/01/2025 01:57:24.777 7f75daa595e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'] not found in DB: Unknown id: access_control_list default-domain:default-project:__no_rule__:ingress-access-control-list 09/01/2025 01:57:24.782 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:ingress-access-control-list url = http://10.0.0.56:8082/access-control-lists operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.56 body = {"access-control-list": {"uuid": null, "fq_name": ["default-domain", "default-project", "__no_rule__", "ingress-access-control-list"], "parent_type": "security-group", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "ingress-access-control-list"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:24.802 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:57:24.792727', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:24.792727', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10277467410935267082, 'uuid_mslong': 16051455917979224025}}, 'parent_type': 'security-group', 'parent_uuid': '620e0e4f-8700-4660-910a-f945774113ec', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'dec23a1c-f6fe-47d9-8ea0-e6286b733f0a'}, 'oper': 'CREATE', 'request-id': 'req-244e0edf-03d4-4c75-962c-14b99f6eb36c', 'type': 'access_control_list', 'uuid': 'dec23a1c-f6fe-47d9-8ea0-e6286b733f0a'} 09/01/2025 01:57:24.859 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'analyzer-template'] not found in DB: Unknown id: service_template default-domain:analyzer-template 09/01/2025 01:57:24.860 7f75da7db900 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:analyzer-template url = http://10.0.0.56:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.56 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "analyzer-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": null, "service_type": "analyzer", "image_name": "analyzer", "service_scaling": false, "interface_type": [{"service_interface_type": "left", "shared_ip": false, "static_route_enable": false}], "flavor": "m1.medium", "ordered_interfaces": true, "service_virtualization_type": "virtual-machine", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "9ef9ee5db15946d4bf11570dab196c5a", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "analyzer-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:24.875 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'analyzer-template'], 'obj_dict': {'display_name': 'analyzer-template', 'fq_name': ['default-domain', 'analyzer-template'], 'id_perms': {'created': '2025-09-01T01:57:24.869959', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:24.869959', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13521180333866150676, 'uuid_mslong': 9793958929396417514}}, 'parent_type': 'domain', 'parent_uuid': '29b069c3-005d-4827-93b3-22f4ba94fd44', 'perms2': {'global_access': 0, 'owner': '9ef9ee5db15946d4bf11570dab196c5a', 'owner_access': 7, 'share': [{'tenant': 'domain:29b069c3-005d-4827-93b3-22f4ba94fd44', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': 'm1.medium', 'image_name': 'analyzer', 'instance_data': None, 'interface_type': [{'service_interface_type': 'left', 'shared_ip': False, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': None, 'service_scaling': False, 'service_type': 'analyzer', 'service_virtualization_type': 'virtual-machine', 'version': 1, 'vrouter_instance_type': None}, 'uuid': '87eb21c1-7370-43ea-bba4-e1e01816eb14'}, 'oper': 'CREATE', 'request-id': 'req-3c8bbc39-dc34-4efc-9dc6-057f1b6d0040', 'type': 'service_template', 'uuid': '87eb21c1-7370-43ea-bba4-e1e01816eb14'} 09/01/2025 01:57:24.883 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'nat-template'] not found in DB: Unknown id: service_template default-domain:nat-template 09/01/2025 01:57:24.884 7f75da7db900 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:nat-template url = http://10.0.0.56:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.56 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "nat-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "in-network-nat", "service_type": "firewall", "image_name": "analyzer", "service_scaling": false, "interface_type": [{"service_interface_type": "management", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "right", "shared_ip": false, "static_route_enable": false}], "flavor": "m1.medium", "ordered_interfaces": true, "service_virtualization_type": "virtual-machine", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "9ef9ee5db15946d4bf11570dab196c5a", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "nat-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:24.892 7f75daa595e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'] not found in DB: Unknown id: access_control_list default-domain:default-project:__no_rule__:egress-access-control-list 09/01/2025 01:57:24.895 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:egress-access-control-list url = http://10.0.0.56:8082/access-control-lists operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.56 body = {"access-control-list": {"uuid": null, "fq_name": ["default-domain", "default-project", "__no_rule__", "egress-access-control-list"], "parent_type": "security-group", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "egress-access-control-list"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:24.913 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'nat-template'], 'obj_dict': {'display_name': 'nat-template', 'fq_name': ['default-domain', 'nat-template'], 'id_perms': {'created': '2025-09-01T01:57:24.905120', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:24.905120', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11611091319806425669, 'uuid_mslong': 5332057076780975597}}, 'parent_type': 'domain', 'parent_uuid': '29b069c3-005d-4827-93b3-22f4ba94fd44', 'perms2': {'global_access': 0, 'owner': '9ef9ee5db15946d4bf11570dab196c5a', 'owner_access': 7, 'share': [{'tenant': 'domain:29b069c3-005d-4827-93b3-22f4ba94fd44', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': 'm1.medium', 'image_name': 'analyzer', 'instance_data': None, 'interface_type': [{'service_interface_type': 'management', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'right', 'shared_ip': False, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'in-network-nat', 'service_scaling': False, 'service_type': 'firewall', 'service_virtualization_type': 'virtual-machine', 'version': 1, 'vrouter_instance_type': None}, 'uuid': '49ff45a9-2fa3-45ed-a122-e20983aefa45'}, 'oper': 'CREATE', 'request-id': 'req-04b84baf-533d-4ec6-a4cc-3aa2efb8f065', 'type': 'service_template', 'uuid': '49ff45a9-2fa3-45ed-a122-e20983aefa45'} 09/01/2025 01:57:24.918 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:57:24.909576', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:24.909576', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12868985776250880682, 'uuid_mslong': 2405709873795646673}}, 'parent_type': 'security-group', 'parent_uuid': '620e0e4f-8700-4660-910a-f945774113ec', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '2162cc62-5c3d-44d1-b297-d26174fd02aa'}, 'oper': 'CREATE', 'request-id': 'req-49824fae-f844-44b8-acce-680508088a94', 'type': 'access_control_list', 'uuid': '2162cc62-5c3d-44d1-b297-d26174fd02aa'} 09/01/2025 01:57:24.921 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:24.970 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'netns-snat-template'] not found in DB: Unknown id: service_template default-domain:netns-snat-template 09/01/2025 01:57:24.972 7f75da7db900 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:netns-snat-template url = http://10.0.0.56:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.56 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "netns-snat-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "in-network-nat", "service_type": "source-nat", "image_name": null, "service_scaling": true, "interface_type": [{"service_interface_type": "right", "shared_ip": true, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": true, "static_route_enable": false}], "flavor": null, "ordered_interfaces": true, "service_virtualization_type": "network-namespace", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "9ef9ee5db15946d4bf11570dab196c5a", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "netns-snat-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:24.996 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'netns-snat-template'], 'obj_dict': {'display_name': 'netns-snat-template', 'fq_name': ['default-domain', 'netns-snat-template'], 'id_perms': {'created': '2025-09-01T01:57:24.988887', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:24.988887', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13127125860065436648, 'uuid_mslong': 9297780873956968404}}, 'parent_type': 'domain', 'parent_uuid': '29b069c3-005d-4827-93b3-22f4ba94fd44', 'perms2': {'global_access': 0, 'owner': '9ef9ee5db15946d4bf11570dab196c5a', 'owner_access': 7, 'share': [{'tenant': 'domain:29b069c3-005d-4827-93b3-22f4ba94fd44', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': None, 'image_name': None, 'instance_data': None, 'interface_type': [{'service_interface_type': 'right', 'shared_ip': True, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': True, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'in-network-nat', 'service_scaling': True, 'service_type': 'source-nat', 'service_virtualization_type': 'network-namespace', 'version': 1, 'vrouter_instance_type': None}, 'uuid': '81085a71-6fa0-47d4-b62c-eb6b530b63e8'}, 'oper': 'CREATE', 'request-id': 'req-e74f30f8-6872-4452-828c-6921aa761df1', 'type': 'service_template', 'uuid': '81085a71-6fa0-47d4-b62c-eb6b530b63e8'} 09/01/2025 01:57:25.024 7f75daa595e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000000'] not found in DB: Unknown id: route_target target:64512:8000000 09/01/2025 01:57:25.025 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000000 url = http://10.0.0.56:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.56 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000000"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "9ef9ee5db15946d4bf11570dab196c5a", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000000"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:25.039 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000000'], 'obj_dict': {'display_name': 'target:64512:8000000', 'fq_name': ['target:64512:8000000'], 'id_perms': {'created': '2025-09-01T01:57:25.032076', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:25.032076', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13445005882208166071, 'uuid_mslong': 3179637496494903218}}, 'perms2': {'global_access': 0, 'owner': '9ef9ee5db15946d4bf11570dab196c5a', 'owner_access': 7, 'share': []}, 'uuid': '2c205774-e517-4fb2-ba96-419d7c0e20b7'}, 'oper': 'CREATE', 'request-id': 'req-62e5a752-f19f-4215-b343-d75b199558ab', 'type': 'route_target', 'uuid': '2c205774-e517-4fb2-ba96-419d7c0e20b7'} 09/01/2025 01:57:25.049 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = e539740b-2375-4404-b14f-3d22969f17f7 object_type = routing_instance identifier_name = default-domain:default-project:dci-network:__default__ url = http://10.0.0.56:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.56 body = {"type": "routing-instance", "uuid": "e539740b-2375-4404-b14f-3d22969f17f7", "ref-type": "route-target", "ref-uuid": "2c205774-e517-4fb2-ba96-419d7c0e20b7", "ref-fq-name": ["target:64512:8000000"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:25.056 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'haproxy-loadbalancer-template'] not found in DB: Unknown id: service_template default-domain:haproxy-loadbalancer-template 09/01/2025 01:57:25.057 7f75da7db900 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:haproxy-loadbalancer-template url = http://10.0.0.56:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.56 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "haproxy-loadbalancer-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "in-network-nat", "service_type": "loadbalancer", "image_name": null, "service_scaling": true, "interface_type": [{"service_interface_type": "right", "shared_ip": true, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": true, "static_route_enable": false}], "flavor": null, "ordered_interfaces": true, "service_virtualization_type": "network-namespace", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "9ef9ee5db15946d4bf11570dab196c5a", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "haproxy-loadbalancer-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:25.066 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'oper': 'UPDATE', 'request-id': 'req-e5e94a13-1fc3-4c2c-9f3d-8a05f4552f88', 'type': 'routing_instance', 'uuid': 'e539740b-2375-4404-b14f-3d22969f17f7'} 09/01/2025 01:57:25.082 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'haproxy-loadbalancer-template'], 'obj_dict': {'display_name': 'haproxy-loadbalancer-template', 'fq_name': ['default-domain', 'haproxy-loadbalancer-template'], 'id_perms': {'created': '2025-09-01T01:57:25.075674', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:25.075674', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9521065237204121824, 'uuid_mslong': 5101078152735048275}}, 'parent_type': 'domain', 'parent_uuid': '29b069c3-005d-4827-93b3-22f4ba94fd44', 'perms2': {'global_access': 0, 'owner': '9ef9ee5db15946d4bf11570dab196c5a', 'owner_access': 7, 'share': [{'tenant': 'domain:29b069c3-005d-4827-93b3-22f4ba94fd44', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': None, 'image_name': None, 'instance_data': None, 'interface_type': [{'service_interface_type': 'right', 'shared_ip': True, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': True, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'in-network-nat', 'service_scaling': True, 'service_type': 'loadbalancer', 'service_virtualization_type': 'network-namespace', 'version': 1, 'vrouter_instance_type': None}, 'uuid': '46caab8d-a147-4253-8421-9e6372df38e0'}, 'oper': 'CREATE', 'request-id': 'req-24055c1d-0d35-48a4-b344-c5fe0f077328', 'type': 'service_template', 'uuid': '46caab8d-a147-4253-8421-9e6372df38e0'} 09/01/2025 01:57:25.137 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'docker-template'] not found in DB: Unknown id: service_template default-domain:docker-template 09/01/2025 01:57:25.138 7f75da7db900 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:docker-template url = http://10.0.0.56:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.56 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "docker-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "transparent", "service_type": "firewall", "image_name": "ubuntu", "service_scaling": false, "interface_type": [{"service_interface_type": "management", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "right", "shared_ip": false, "static_route_enable": false}], "flavor": null, "ordered_interfaces": true, "service_virtualization_type": "vrouter-instance", "availability_zone_enable": false, "vrouter_instance_type": "docker", "instance_data": "{\"command\":\"/bin/bash\"}"}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "9ef9ee5db15946d4bf11570dab196c5a", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "docker-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:25.143 7f75daa595e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000001'] not found in DB: Unknown id: route_target target:64512:8000001 09/01/2025 01:57:25.145 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000001 url = http://10.0.0.56:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.56 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000001"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "9ef9ee5db15946d4bf11570dab196c5a", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000001"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:25.175 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'docker-template'], 'obj_dict': {'display_name': 'docker-template', 'fq_name': ['default-domain', 'docker-template'], 'id_perms': {'created': '2025-09-01T01:57:25.159671', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:25.159671', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11139179131399977529, 'uuid_mslong': 8789539564019075591}}, 'parent_type': 'domain', 'parent_uuid': '29b069c3-005d-4827-93b3-22f4ba94fd44', 'perms2': {'global_access': 0, 'owner': '9ef9ee5db15946d4bf11570dab196c5a', 'owner_access': 7, 'share': [{'tenant': 'domain:29b069c3-005d-4827-93b3-22f4ba94fd44', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': None, 'image_name': 'ubuntu', 'instance_data': '{"command":"/bin/bash"}', 'interface_type': [{'service_interface_type': 'management', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'right', 'shared_ip': False, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'transparent', 'service_scaling': False, 'service_type': 'firewall', 'service_virtualization_type': 'vrouter-instance', 'version': 1, 'vrouter_instance_type': 'docker'}, 'uuid': '79fab7aa-1c99-4a07-9a96-5066f08a3a39'}, 'oper': 'CREATE', 'request-id': 'req-a15d2453-ecec-4c90-b53a-1fbff95e0c87', 'type': 'service_template', 'uuid': '79fab7aa-1c99-4a07-9a96-5066f08a3a39'} 09/01/2025 01:57:25.177 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000001'], 'obj_dict': {'display_name': 'target:64512:8000001', 'fq_name': ['target:64512:8000001'], 'id_perms': {'created': '2025-09-01T01:57:25.162274', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:25.162274', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13507925698563971393, 'uuid_mslong': 3087026659819275195}}, 'perms2': {'global_access': 0, 'owner': '9ef9ee5db15946d4bf11570dab196c5a', 'owner_access': 7, 'share': []}, 'uuid': '2ad75264-2eb0-43bb-bb75-cadad3469941'}, 'oper': 'CREATE', 'request-id': 'req-00ff42a2-3e12-4da3-8957-ea788e904778', 'type': 'route_target', 'uuid': '2ad75264-2eb0-43bb-bb75-cadad3469941'} 09/01/2025 01:57:25.225 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 8c7a11ed-d168-41be-b4f9-cd3a9b2f8025 object_type = routing_instance identifier_name = default-domain:default-project:dci-network:dci-network url = http://10.0.0.56:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.56 body = {"type": "routing-instance", "uuid": "8c7a11ed-d168-41be-b4f9-cd3a9b2f8025", "ref-type": "route-target", "ref-uuid": "2ad75264-2eb0-43bb-bb75-cadad3469941", "ref-fq-name": ["target:64512:8000001"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:25.237 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'oper': 'UPDATE', 'request-id': 'req-0ae7a961-afa0-41a0-8773-c73ae67701ea', 'type': 'routing_instance', 'uuid': '8c7a11ed-d168-41be-b4f9-cd3a9b2f8025'} 09/01/2025 01:57:25.311 7f75daa595e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000002'] not found in DB: Unknown id: route_target target:64512:8000002 09/01/2025 01:57:25.312 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000002 url = http://10.0.0.56:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.56 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000002"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "9ef9ee5db15946d4bf11570dab196c5a", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000002"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:25.333 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000002'], 'obj_dict': {'display_name': 'target:64512:8000002', 'fq_name': ['target:64512:8000002'], 'id_perms': {'created': '2025-09-01T01:57:25.321933', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:25.321933', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10854995415371099679, 'uuid_mslong': 3718999023607695746}}, 'perms2': {'global_access': 0, 'owner': '9ef9ee5db15946d4bf11570dab196c5a', 'owner_access': 7, 'share': []}, 'uuid': '339c89e8-6a29-4982-96a4-b0d16bb02e1f'}, 'oper': 'CREATE', 'request-id': 'req-21ed0f20-9ef0-49ea-b63a-d952189ced06', 'type': 'route_target', 'uuid': '339c89e8-6a29-4982-96a4-b0d16bb02e1f'} 09/01/2025 01:57:25.385 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = c5187ec2-16a9-4faa-b7ba-5b40f765ed82 object_type = routing_instance identifier_name = default-domain:default-project:default-virtual-network:default-virtual-network url = http://10.0.0.56:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.56 body = {"type": "routing-instance", "uuid": "c5187ec2-16a9-4faa-b7ba-5b40f765ed82", "ref-type": "route-target", "ref-uuid": "339c89e8-6a29-4982-96a4-b0d16bb02e1f", "ref-fq-name": ["target:64512:8000002"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:25.400 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'oper': 'UPDATE', 'request-id': 'req-c36cfde0-2fa3-4f19-920d-5e7ab8ac023e', 'type': 'routing_instance', 'uuid': 'c5187ec2-16a9-4faa-b7ba-5b40f765ed82'} 09/01/2025 01:57:25.462 7f75daa595e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000003'] not found in DB: Unknown id: route_target target:64512:8000003 09/01/2025 01:57:25.463 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000003 url = http://10.0.0.56:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.56 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000003"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "9ef9ee5db15946d4bf11570dab196c5a", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000003"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:25.480 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000003'], 'obj_dict': {'display_name': 'target:64512:8000003', 'fq_name': ['target:64512:8000003'], 'id_perms': {'created': '2025-09-01T01:57:25.471548', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:25.471548', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9247724033477844785, 'uuid_mslong': 6465635102301045560}}, 'perms2': {'global_access': 0, 'owner': '9ef9ee5db15946d4bf11570dab196c5a', 'owner_access': 7, 'share': []}, 'uuid': '59ba8cde-7ff3-4338-8056-84030c4a1f31'}, 'oper': 'CREATE', 'request-id': 'req-65150cd0-0e7a-46ef-a203-0daabcc51b27', 'type': 'route_target', 'uuid': '59ba8cde-7ff3-4338-8056-84030c4a1f31'} 09/01/2025 01:57:25.530 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 231b437c-f8ca-4984-8584-b66c8b6969d7 object_type = routing_instance identifier_name = default-domain:default-project:ip-fabric:ip-fabric url = http://10.0.0.56:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.56 body = {"type": "routing-instance", "uuid": "231b437c-f8ca-4984-8584-b66c8b6969d7", "ref-type": "route-target", "ref-uuid": "59ba8cde-7ff3-4338-8056-84030c4a1f31", "ref-fq-name": ["target:64512:8000003"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:25.546 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-6be96408-0b1d-4ad3-b461-d00a13833cec', 'type': 'routing_instance', 'uuid': '231b437c-f8ca-4984-8584-b66c8b6969d7'} 09/01/2025 01:57:25.641 7f75daa595e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'] not found in DB: Unknown id: access_control_list default-domain:default-project:ip-fabric:ip-fabric 09/01/2025 01:57:25.646 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = access_control_list identifier_name = default-domain:default-project:ip-fabric:ip-fabric url = http://10.0.0.56:8082/access-control-lists operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.56 body = {"access-control-list": {"uuid": null, "fq_name": ["default-domain", "default-project", "ip-fabric", "ip-fabric"], "parent_type": "virtual-network", "access_control_list_entries": {"dynamic": false, "acl_rule": [{"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "deny", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000002", "direction": "<>"}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "ip-fabric"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:25.714 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'deny'}, 'direction': '<>', 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000002'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}], 'dynamic': False}, 'display_name': 'ip-fabric', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'id_perms': {'created': '2025-09-01T01:57:25.659948', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:25.659948', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9366827408060485335, 'uuid_mslong': 7610566125744046283}}, 'parent_type': 'virtual-network', 'parent_uuid': 'c39bc5a7-3ad4-482b-9be0-1adbb0ac1242', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '699e2991-a098-40cb-81fd-a7e669d68ed7'}, 'oper': 'CREATE', 'request-id': 'req-2a764eae-e604-4a32-8250-fec835f9fa09', 'type': 'access_control_list', 'uuid': '699e2991-a098-40cb-81fd-a7e669d68ed7'} 09/01/2025 01:57:25.768 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = dec23a1c-f6fe-47d9-8ea0-e6286b733f0a object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:ingress-access-control-list url = http://10.0.0.56:8082/access-control-list/dec23a1c-f6fe-47d9-8ea0-e6286b733f0a operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.56 body = {"access-control-list": {"uuid": "dec23a1c-f6fe-47d9-8ea0-e6286b733f0a", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "access_control_list_hash": 3520976145078303111}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:25.785 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-dccc92b7-b6c9-4f5a-beea-2caa11556305', 'type': 'access_control_list', 'uuid': 'dec23a1c-f6fe-47d9-8ea0-e6286b733f0a'} 09/01/2025 01:57:25.835 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 2162cc62-5c3d-44d1-b297-d26174fd02aa object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:egress-access-control-list url = http://10.0.0.56:8082/access-control-list/2162cc62-5c3d-44d1-b297-d26174fd02aa operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.56 body = {"access-control-list": {"uuid": "2162cc62-5c3d-44d1-b297-d26174fd02aa", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "access_control_list_hash": 3520976145078303111}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:25.853 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-eb120720-53d3-4802-9f39-e69ba89416d2', 'type': 'access_control_list', 'uuid': '2162cc62-5c3d-44d1-b297-d26174fd02aa'} 09/01/2025 01:57:25.922 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:26.924 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:27.868 7f75da7db540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:27.870 7f75da7db540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:27.871 7f75da7db540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:27.871 7f75da7db540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:27.871 7f75da7db540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:27.872 7f75da7db540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:27.872 7f75da7db540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:27.873 7f75da7db540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:27.873 7f75da7db540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:27.873 7f75da7db540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:27.874 7f75da7db540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:27.874 7f75da7db540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:27.874 7f75da7db540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:27.874 7f75da7db540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:27.926 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:28.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:29.151 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_objects in resource global-vrouter-config: 'security_logging_objects' 09/01/2025 01:57:29.151 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-vrouter-config: 'tag_refs' 09/01/2025 01:57:29.152 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field application_policy_set_back_refs in resource global-vrouter-config: 'application_policy_set_back_refs' 09/01/2025 01:57:29.262 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 08473562-4927-44da-a90c-47622528dc25 object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.56:8082/global-vrouter-config/08473562-4927-44da-a90c-47622528dc25 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_linklocal.py remote_ip = 10.0.0.56 body = {"global-vrouter-config": {"uuid": "08473562-4927-44da-a90c-47622528dc25", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "linklocal_services": {"linklocal_service_entry": [{"linklocal_service_name": "metadata", "linklocal_service_ip": "169.254.169.254", "linklocal_service_port": 80, "ip_fabric_DNS_service_name": "", "ip_fabric_service_port": 8775, "ip_fabric_service_ip": ["10.0.0.56"]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:29.279 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-cab338b1-5405-4a03-a3a6-05c2a292d4be', 'type': 'global_vrouter_config', 'uuid': '08473562-4927-44da-a90c-47622528dc25'} 09/01/2025 01:57:29.929 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:30.028 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-conf-elements-inconsistency'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-conf-elements-inconsistency 09/01/2025 01:57:30.036 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-conf-elements-inconsistency url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-conf-elements-inconsistency"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements", "operation": "size==", "operand2": {"json_value": "1"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "ContrailConfig has a list of elements - it should be a dict. Controllers have inconsistency in their data.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-conf-elements-inconsistency"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:30.072 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements', 'operand2': {'json_value': '1'}, 'operation': 'size=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-elements-inconsistency', 'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'id_perms': {'created': '2025-09-01T01:57:30.054199', 'creator': None, 'description': 'ContrailConfig has a list of ' 'elements - it should be a dict. ' 'Controllers have inconsistency in ' 'their data.', 'enable': True, 'last_modified': '2025-09-01T01:57:30.054199', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9464576858265937879, 'uuid_mslong': 3056869154938963818}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '2a6c2e4c-e8fc-476a-8358-ee81d252e7d7', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-b2bae1b0-7f80-473e-ab04-39c4345dfca8', 'type': 'alarm', 'uuid': '2a6c2e4c-e8fc-476a-8358-ee81d252e7d7'} 09/01/2025 01:57:30.075 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-address-mismatch-compute'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-address-mismatch-compute 09/01/2025 01:57:30.084 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-address-mismatch-compute url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-address-mismatch-compute"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_ip_address", "operation": "!=", "operand2": {"uve_attribute": "VrouterAgent.control_ip"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Compute Node IP Address mismatch.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-address-mismatch-compute"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:30.117 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_ip_address', 'operand2': {'uve_attribute': 'VrouterAgent.control_ip'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-compute', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'id_perms': {'created': '2025-09-01T01:57:30.104070', 'creator': None, 'description': 'Compute Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:57:30.104070', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9683332200319043821, 'uuid_mslong': 1251073803591502090}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '115cb4fe-f2de-490a-8662-1b54ea1e48ed', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-5dc330d9-03e7-43aa-a461-4faac79b70b6', 'type': 'alarm', 'uuid': '115cb4fe-f2de-490a-8662-1b54ea1e48ed'} 09/01/2025 01:57:30.166 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-address-mismatch-control'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-address-mismatch-control 09/01/2025 01:57:30.175 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-address-mismatch-control url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-address-mismatch-control"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.bgp_router_parameters.address", "operation": "not in", "operand2": {"uve_attribute": "BgpRouterState.bgp_router_ip_list"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Control Node IP Address mismatch.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-address-mismatch-control"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:30.200 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.bgp_router_parameters.address', 'operand2': {'uve_attribute': 'BgpRouterState.bgp_router_ip_list'}, 'operation': 'not ' 'in'}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-control', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'id_perms': {'created': '2025-09-01T01:57:30.193851', 'creator': None, 'description': 'Control Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:57:30.193851', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10830535933819907698, 'uuid_mslong': 3553504101098670032}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '31509527-15e1-47d0-964d-cb0c96706e72', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-b6efde5a-c06e-454f-befa-982ca572b141', 'type': 'alarm', 'uuid': '31509527-15e1-47d0-964d-cb0c96706e72'} 09/01/2025 01:57:30.249 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-bgp-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-bgp-connectivity 09/01/2025 01:57:30.257 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-bgp-connectivity url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-bgp-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "BgpRouterState.num_up_bgp_peer", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "BgpRouterState.num_up_bgp_peer", "operation": "!=", "operand2": {"uve_attribute": "BgpRouterState.num_bgp_peer"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "BGP peer mismatch. Not enough BGP peers are up.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-bgp-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:30.293 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_bgp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-bgp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:30.275956', 'creator': None, 'description': 'BGP peer mismatch. Not enough BGP ' 'peers are up.', 'enable': True, 'last_modified': '2025-09-01T01:57:30.275956', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13395938943348192093, 'uuid_mslong': 14326837012404456381}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c6d3265b-1f49-4bbd-b9e7-ef7e8717335d', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-f19c69de-e327-4ad3-8f80-7882fc437403', 'type': 'alarm', 'uuid': 'c6d3265b-1f49-4bbd-b9e7-ef7e8717335d'} 09/01/2025 01:57:30.341 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-conf-incorrect'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-conf-incorrect 09/01/2025 01:57:30.348 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-conf-incorrect url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-conf-incorrect"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig", "operation": "==", "operand2": {"json_value": "null"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-conf-incorrect"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:30.377 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-incorrect', 'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'id_perms': {'created': '2025-09-01T01:57:30.367088', 'creator': None, 'description': 'ContrailConfig missing or ' 'incorrect. Configuration pushed to ' 'Ifmap as ContrailConfig is ' 'missing/incorrect.', 'enable': True, 'last_modified': '2025-09-01T01:57:30.367088', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10820269243797746688, 'uuid_mslong': 15186166341629398445}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd2c019d3-f4d0-4dad-9629-518c8aa63400', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-9da8650e-2b6a-4265-91fe-cd164a42b888', 'type': 'alarm', 'uuid': 'd2c019d3-f4d0-4dad-9629-518c8aa63400'} 09/01/2025 01:57:30.424 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-disk-usage-high'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-disk-usage-high 09/01/2025 01:57:30.429 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-disk-usage-high url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-disk-usage-high"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operation": "range", "operand2": {"json_value": "[70, 90]"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Disk usage crosses high threshold limit.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-disk-usage-high"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:30.460 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '[70, ' '90]'}, 'operation': 'range', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-disk-usage-high', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'id_perms': {'created': '2025-09-01T01:57:30.447796', 'creator': None, 'description': 'Disk usage crosses high threshold ' 'limit.', 'enable': True, 'last_modified': '2025-09-01T01:57:30.447796', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10351272032287242206, 'uuid_mslong': 6248005955484206403}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '56b56050-2b29-4543-8fa7-1b11e9eab7de', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-885a4a39-2518-482b-bd47-96515700cec2', 'type': 'alarm', 'uuid': '56b56050-2b29-4543-8fa7-1b11e9eab7de'} 09/01/2025 01:57:30.505 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-disk-usage-critical'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-disk-usage-critical 09/01/2025 01:57:30.512 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-disk-usage-critical url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-disk-usage-critical"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operation": ">", "operand2": {"json_value": "90"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Disk usage crosses critical threshold limit.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-disk-usage-critical"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:30.552 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '90'}, 'operation': '>', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-disk-usage-critical', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'id_perms': {'created': '2025-09-01T01:57:30.537335', 'creator': None, 'description': 'Disk usage crosses critical ' 'threshold limit.', 'enable': True, 'last_modified': '2025-09-01T01:57:30.537335', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13600940164448598912, 'uuid_mslong': 4838096181709131050}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '43245ed8-0bf2-4d2a-bcc0-3f04c94d3b80', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-f99e466d-c4b1-4502-8a72-df965eec5c58', 'type': 'alarm', 'uuid': '43245ed8-0bf2-4d2a-bcc0-3f04c94d3b80'} 09/01/2025 01:57:30.601 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-node-status'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-node-status 09/01/2025 01:57:30.609 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-node-status url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-node-status"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus", "operation": "==", "operand2": {"json_value": "null"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Node Failure. NodeStatus UVE not present.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-node-status"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:30.640 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-node-status', 'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'id_perms': {'created': '2025-09-01T01:57:30.628902', 'creator': None, 'description': 'Node Failure. NodeStatus UVE not ' 'present.', 'enable': True, 'last_modified': '2025-09-01T01:57:30.628902', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10735481443780481004, 'uuid_mslong': 9153565873247047496}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '7f07ffa9-ff07-4f48-94fc-177e7d554fec', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-9d9d857a-e9be-437d-b6de-a49eb9a09c6f', 'type': 'alarm', 'uuid': '7f07ffa9-ff07-4f48-94fc-177e7d554fec'} 09/01/2025 01:57:30.689 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-partial-sysinfo'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-partial-sysinfo 09/01/2025 01:57:30.696 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-partial-sysinfo url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-partial-sysinfo"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.build_info", "operation": "==", "operand2": {"json_value": "null"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "System Info Incomplete.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-partial-sysinfo"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:30.727 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.build_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-partial-sysinfo', 'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'id_perms': {'created': '2025-09-01T01:57:30.714837', 'creator': None, 'description': 'System Info Incomplete.', 'enable': True, 'last_modified': '2025-09-01T01:57:30.714837', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11150891449519288517, 'uuid_mslong': 14343575021455294730}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c70e9d7c-8b02-410a-9abf-ecb18189b0c5', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-5a272799-5d50-4f04-8b7d-e4da887cec37', 'type': 'alarm', 'uuid': 'c70e9d7c-8b02-410a-9abf-ecb18189b0c5'} 09/01/2025 01:57:30.775 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-process-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-process-connectivity 09/01/2025 01:57:30.783 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-process-connectivity url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-process-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.process_status", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "NodeStatus.process_status.state", "operation": "!=", "operand2": {"json_value": "\"Functional\""}, "variables": ["NodeStatus.process_status.module_id", "NodeStatus.process_status.instance_id"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Process(es) reporting as non-functional.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-process-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:30.818 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_status', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_status.state', 'operand2': {'json_value': '"Functional"'}, 'operation': '!=', 'variables': ['NodeStatus.process_status.module_id', 'NodeStatus.process_status.instance_id']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:30.803175', 'creator': None, 'description': 'Process(es) reporting as ' 'non-functional.', 'enable': True, 'last_modified': '2025-09-01T01:57:30.803175', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10894851833224389702, 'uuid_mslong': 8418205608933870664}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '74d37966-56b4-4c48-9732-4a04d7da6446', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-f13093c2-2e62-4fa4-b04b-790d9294fda9', 'type': 'alarm', 'uuid': '74d37966-56b4-4c48-9732-4a04d7da6446'} 09/01/2025 01:57:30.865 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-process-status'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-process-status 09/01/2025 01:57:30.873 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-process-status url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-process-status"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.process_info", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "NodeStatus.process_info.process_state", "operation": "!=", "operand2": {"json_value": "\"PROCESS_STATE_RUNNING\""}, "variables": ["NodeStatus.process_info.process_name"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Process Failure.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-process-status"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:30.905 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_info.process_state', 'operand2': {'json_value': '"PROCESS_STATE_RUNNING"'}, 'operation': '!=', 'variables': ['NodeStatus.process_info.process_name']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-status', 'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'id_perms': {'created': '2025-09-01T01:57:30.892287', 'creator': None, 'description': 'Process Failure.', 'enable': True, 'last_modified': '2025-09-01T01:57:30.892287', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13607781469830687097, 'uuid_mslong': 1493426681964743480}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '14b9b7a7-9c99-4b38-bcd8-8d265266a979', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-24782ce6-af27-43f3-9247-e61e3244ad38', 'type': 'alarm', 'uuid': '14b9b7a7-9c99-4b38-bcd8-8d265266a979'} 09/01/2025 01:57:30.931 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:30.956 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-prouter-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-prouter-connectivity 09/01/2025 01:57:30.962 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-prouter-connectivity url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-prouter-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["prouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.connected_agent_list", "operation": "size!=", "operand2": {"json_value": "1"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Prouter connectivity to controlling tor agent does not exist we look for non-empty value for connected_agent_list", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-prouter-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:30.992 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.connected_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:30.980556', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tor agent does not exist we look ' 'for non-empty value for ' 'connected_agent_list', 'enable': True, 'last_modified': '2025-09-01T01:57:30.980556', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10798366937868962885, 'uuid_mslong': 11696224369911022768}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a2515618-5dd0-44b0-95db-81848e008c45', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-444639b2-25b0-42ae-9634-492d50469689', 'type': 'alarm', 'uuid': 'a2515618-5dd0-44b0-95db-81848e008c45'} 09/01/2025 01:57:31.035 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-prouter-tsn-connectivity 09/01/2025 01:57:31.044 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-prouter-tsn-connectivity url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-prouter-tsn-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["prouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.tsn_agent_list", "operation": "size!=", "operand2": {"json_value": "1"}}, {"operand1": "ProuterData.gateway_mode", "operation": "!=", "operand2": {"json_value": "\"SERVER\""}}]}, {"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.tsn_agent_list", "operation": "size!=", "operand2": {"json_value": "0"}}, {"operand1": "ProuterData.gateway_mode", "operation": "==", "operand2": {"json_value": "\"SERVER\""}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Prouter connectivity to controlling tsn agent does not exist we look for non-empty value for tsn_agent_list", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-prouter-tsn-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:31.116 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '!='}]}, {'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-tsn-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:31.058068', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tsn agent does not exist we look ' 'for non-empty value for ' 'tsn_agent_list', 'enable': True, 'last_modified': '2025-09-01T01:57:31.058068', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9679409398720970998, 'uuid_mslong': 7924194410190816163}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6df864d4-f68f-43a3-8654-2b907e5740f6', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-7169866a-210c-4724-9b3f-4a8aad656db4', 'type': 'alarm', 'uuid': '6df864d4-f68f-43a3-8654-2b907e5740f6'} 09/01/2025 01:57:31.160 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-storage-cluster-state'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-storage-cluster-state 09/01/2025 01:57:31.167 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-storage-cluster-state url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-storage-cluster-state"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["storage-cluster"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "StorageCluster.info_stats.status", "operation": "!=", "operand2": {"json_value": "0"}, "variables": ["StorageCluster.info_stats.health_summary"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Storage Cluster warning/errors.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-storage-cluster-state"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:31.203 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'StorageCluster.info_stats.status', 'operand2': {'json_value': '0'}, 'operation': '!=', 'variables': ['StorageCluster.info_stats.health_summary']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-storage-cluster-state', 'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'id_perms': {'created': '2025-09-01T01:57:31.187380', 'creator': None, 'description': 'Storage Cluster warning/errors.', 'enable': True, 'last_modified': '2025-09-01T01:57:31.187380', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10316165217059830525, 'uuid_mslong': 6890667409092331147}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5fa09180-104f-468b-8f2a-619b87177afd', 'uve_keys': {'uve_key': ['storage-cluster']}}, 'oper': 'CREATE', 'request-id': 'req-6daa7478-8fba-48d9-8729-cfa776384766', 'type': 'alarm', 'uuid': '5fa09180-104f-468b-8f2a-619b87177afd'} 09/01/2025 01:57:31.248 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-interface'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-interface 09/01/2025 01:57:31.256 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-interface url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-interface"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.down_interface_count", "operation": ">=", "operand2": {"json_value": "1"}, "variables": ["VrouterAgent.error_intf_list", "VrouterAgent.no_config_intf_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Vrouter interface(s) down.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-interface"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:31.287 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.down_interface_count', 'operand2': {'json_value': '1'}, 'operation': '>=', 'variables': ['VrouterAgent.error_intf_list', 'VrouterAgent.no_config_intf_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-interface', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'id_perms': {'created': '2025-09-01T01:57:31.275962', 'creator': None, 'description': 'Vrouter interface(s) down.', 'enable': True, 'last_modified': '2025-09-01T01:57:31.275962', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11276380950649131098, 'uuid_mslong': 4639482557382018409}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4062c0ce-0e60-4569-9c7d-c0bad36c505a', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-a1360b9c-e102-4f4d-ba6a-b36ab707f172', 'type': 'alarm', 'uuid': '4062c0ce-0e60-4569-9c7d-c0bad36c505a'} 09/01/2025 01:57:31.341 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-xmpp-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-xmpp-connectivity 09/01/2025 01:57:31.349 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-xmpp-connectivity url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-xmpp-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "BgpRouterState.num_up_xmpp_peer", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "BgpRouterState.num_up_xmpp_peer", "operation": "!=", "operand2": {"uve_attribute": "BgpRouterState.num_xmpp_peer"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "XMPP peer mismatch.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-xmpp-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:31.384 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_xmpp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:31.369979', 'creator': None, 'description': 'XMPP peer mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:57:31.369979', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11934772505184287857, 'uuid_mslong': 5811969849151538316}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '50a843c8-dff5-448c-a5a0-d45c3e9bbc71', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-d9a4961b-946b-46bc-a3e9-8e13e1b3565b', 'type': 'alarm', 'uuid': '50a843c8-dff5-448c-a5a0-d45c3e9bbc71'} 09/01/2025 01:57:31.429 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-xmpp-close-reason'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-xmpp-close-reason 09/01/2025 01:57:31.437 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-xmpp-close-reason url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-xmpp-close-reason"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["xmpp-peer"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "XmppPeerInfoData.close_reason", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "XmppPeerInfoData.state_info.state", "operation": "!=", "operand2": {"json_value": "\"Established\""}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "XMPP connection closed towards peer, alarm has close reason", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-xmpp-close-reason"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:31.467 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'XmppPeerInfoData.close_reason', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'XmppPeerInfoData.state_info.state', 'operand2': {'json_value': '"Established"'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-close-reason', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'id_perms': {'created': '2025-09-01T01:57:31.455463', 'creator': None, 'description': 'XMPP connection closed towards ' 'peer, ' 'alarm has close reason', 'enable': True, 'last_modified': '2025-09-01T01:57:31.455463', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10114706536922320676, 'uuid_mslong': 968344522386194724}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0d704036-6d06-4124-8c5e-a8015c938f24', 'uve_keys': {'uve_key': ['xmpp-peer']}}, 'oper': 'CREATE', 'request-id': 'req-8f498e8a-e49f-46ed-ae3b-3f74f6da992d', 'type': 'alarm', 'uuid': '0d704036-6d06-4124-8c5e-a8015c938f24'} 09/01/2025 01:57:31.516 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-core-files'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-core-files 09/01/2025 01:57:31.525 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-core-files url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-core-files"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.all_core_file_list", "operand2": {"json_value": "null"}, "operation": "!="}, {"operand1": "NodeStatus.all_core_file_list", "operand2": {"json_value": "0"}, "operation": "size!="}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "A core file has been generated on the node.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-core-files"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:31.553 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-core-files', 'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'id_perms': {'created': '2025-09-01T01:57:31.544086', 'creator': None, 'description': 'A core file has been generated on ' 'the node.', 'enable': True, 'last_modified': '2025-09-01T01:57:31.544086', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10067795827070312262, 'uuid_mslong': 18329283790498910002}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'fe5eb27e-a803-4f32-8bb7-fef697af9f46', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-1b360d25-4fb4-45e1-8897-c8f7ff7cbe8c', 'type': 'alarm', 'uuid': 'fe5eb27e-a803-4f32-8bb7-fef697af9f46'} 09/01/2025 01:57:31.600 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-pending-cassandra-compaction-tasks 09/01/2025 01:57:31.604 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-pending-cassandra-compaction-tasks url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-pending-cassandra-compaction-tasks"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["database-nodeconfig-database-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks", "operand2": {"json_value": "300"}, "operation": ">="}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Pending compaction tasks in cassandra crossed the configured threshold.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-pending-cassandra-compaction-tasks"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:31.625 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks', 'operand2': {'json_value': '300'}, 'operation': '>='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-pending-cassandra-compaction-tasks', 'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'id_perms': {'created': '2025-09-01T01:57:31.616211', 'creator': None, 'description': 'Pending compaction tasks in ' 'cassandra crossed the configured ' 'threshold.', 'enable': True, 'last_modified': '2025-09-01T01:57:31.616211', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10239805476498944643, 'uuid_mslong': 11871788326234899637}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a4c11095-51dc-44b5-8e1b-18d3fbd1b283', 'uve_keys': {'uve_key': ['database-nodeconfig-database-node']}}, 'oper': 'CREATE', 'request-id': 'req-549d826e-6be4-4c14-8a1b-1b49be709170', 'type': 'alarm', 'uuid': 'a4c11095-51dc-44b5-8e1b-18d3fbd1b283'} 09/01/2025 01:57:31.674 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-package-version-mismatch'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-package-version-mismatch 09/01/2025 01:57:31.682 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-package-version-mismatch url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-package-version-mismatch"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.running_package_version", "operation": "!=", "operand2": {"uve_attribute": "NodeStatus.installed_package_version"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "There is a mismatch between installed and running package version.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-package-version-mismatch"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:31.713 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.running_package_version', 'operand2': {'uve_attribute': 'NodeStatus.installed_package_version'}, 'operation': '!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-package-version-mismatch', 'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'id_perms': {'created': '2025-09-01T01:57:31.699790', 'creator': None, 'description': 'There is a mismatch between ' 'installed and running package ' 'version.', 'enable': True, 'last_modified': '2025-09-01T01:57:31.699790', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11974809962785940499, 'uuid_mslong': 18005816803787425937}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f9e182fb-f082-4891-a62f-123744aa2413', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-b25c7652-604a-42e9-a321-b2e29ed96e83', 'type': 'alarm', 'uuid': 'f9e182fb-f082-4891-a62f-123744aa2413'} 09/01/2025 01:57:31.760 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-limit-exceeded 09/01/2025 01:57:31.769 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-limit-exceeded url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-limit-exceeded"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.res_limit", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Agent resource usage exceeded configured watermark for resource.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-limit-exceeded"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:31.802 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'id_perms': {'created': '2025-09-01T01:57:31.788067', 'creator': None, 'description': 'Agent resource usage exceeded ' 'configured watermark for resource.', 'enable': True, 'last_modified': '2025-09-01T01:57:31.788067', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9724180224848359593, 'uuid_mslong': 4300958671510458183}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '3bb0131f-d104-4f47-86f3-3a64c1f6f0a9', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-8c140382-5c5e-4272-84e6-3fca9df476a0', 'type': 'alarm', 'uuid': '3bb0131f-d104-4f47-86f3-3a64c1f6f0a9'} 09/01/2025 01:57:31.849 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-table-limit-exceeded 09/01/2025 01:57:31.857 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-table-limit-exceeded url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-table-limit-exceeded"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.res_table_limit", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Agent resource usage exceeded table size for resource in vrouter.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-table-limit-exceeded"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:31.871 7f75da7dbf40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:31.875 7f75da7dbf40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:31.875 7f75da7dbf40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:31.875 7f75da7dbf40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:31.876 7f75da7dbf40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:31.876 7f75da7dbf40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:31.876 7f75da7dbf40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:31.877 7f75da7dbf40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:31.877 7f75da7dbf40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:31.878 7f75da7dbf40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:31.878 7f75da7dbf40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:31.878 7f75da7dbf40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:31.878 7f75da7dbf40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:31.878 7f75da7dbf40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:31.894 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_table_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-table-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'id_perms': {'created': '2025-09-01T01:57:31.880047', 'creator': None, 'description': 'Agent resource usage exceeded table ' 'size for resource in vrouter.', 'enable': True, 'last_modified': '2025-09-01T01:57:31.880047', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10601837199416426449, 'uuid_mslong': 13230187526605258878}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b79b1175-7b1c-407e-9321-4ac30dbcffd1', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-b61914d6-6fbc-49d5-971f-11aa0f05175f', 'type': 'alarm', 'uuid': 'b79b1175-7b1c-407e-9321-4ac30dbcffd1'} 09/01/2025 01:57:31.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:31.945 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down 09/01/2025 01:57:31.954 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down url = http://10.0.0.56:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.56 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-l3mh-physical-interface-link-down"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.l3mh_phy_interface_down", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_l3mh_intf_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "On l3 multihomed compute one or more physical interfaces are Inactive.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-l3mh-physical-interface-link-down"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:31.985 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.l3mh_phy_interface_down', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_l3mh_intf_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-l3mh-physical-interface-link-down', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'id_perms': {'created': '2025-09-01T01:57:31.973494', 'creator': None, 'description': 'On l3 multihomed compute one or ' 'more physical interfaces are ' 'Inactive.', 'enable': True, 'last_modified': '2025-09-01T01:57:31.973494', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12950168648613563420, 'uuid_mslong': 4019893145475239651}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '37c98784-7b14-4ee3-b3b8-3dc5e2030c1c', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-4e87efb4-b197-4427-94e9-d8685451095c', 'type': 'alarm', 'uuid': '37c98784-7b14-4ee3-b3b8-3dc5e2030c1c'} 09/01/2025 01:57:32.819 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 08473562-4927-44da-a90c-47622528dc25 object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.56:8082/global-vrouter-config/08473562-4927-44da-a90c-47622528dc25 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_encap.py remote_ip = 10.0.0.56 body = {"global-vrouter-config": {"uuid": "08473562-4927-44da-a90c-47622528dc25", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "encapsulation_priorities": {"encapsulation": ["MPLSoUDP", "MPLSoGRE", "VXLAN"]}, "vxlan_network_identifier_mode": "automatic"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:32.835 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-075165f7-503b-4338-92f4-8865a97d36c7', 'type': 'global_vrouter_config', 'uuid': '08473562-4927-44da-a90c-47622528dc25'} 09/01/2025 01:57:32.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:33.606 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_objects in resource global-vrouter-config: 'security_logging_objects' 09/01/2025 01:57:33.607 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-vrouter-config: 'tag_refs' 09/01/2025 01:57:33.607 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field application_policy_set_back_refs in resource global-vrouter-config: 'application_policy_set_back_refs' 09/01/2025 01:57:33.716 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 08473562-4927-44da-a90c-47622528dc25 object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.56:8082/global-vrouter-config/08473562-4927-44da-a90c-47622528dc25 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_global_vrouter_config.py remote_ip = 10.0.0.56 body = {"global-vrouter-config": {"uuid": "08473562-4927-44da-a90c-47622528dc25", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "port_translation_pools": {"port_translation_pool": []}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:33.741 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-1d308f62-bab3-411e-adcb-b4572e26b8ed', 'type': 'global_vrouter_config', 'uuid': '08473562-4927-44da-a90c-47622528dc25'} 09/01/2025 01:57:33.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:34.561 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:34.562 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:34.562 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:34.562 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:34.562 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:34.563 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:34.563 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:34.563 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:34.563 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:34.563 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:34.564 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:34.564 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:34.564 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:34.564 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:34.565 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:34.565 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:34.565 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:34.565 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:34.566 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:34.566 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:34.566 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:34.566 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:34.566 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:34.567 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:34.567 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:34.567 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:34.567 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:34.567 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:34.568 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:34.568 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:34.568 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:34.568 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:34.568 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:57:34.569 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:34.569 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:34.624 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 7eedf3ea-84b7-4623-96cf-7fecf55ca500 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.56:8082/global-system-config/7eedf3ea-84b7-4623-96cf-7fecf55ca500 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.56 body = {"global-system-config": {"uuid": "7eedf3ea-84b7-4623-96cf-7fecf55ca500", "autonomous_system": 64512, "ibgp_auto_mesh": true}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:34.659 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-b895319b-a6dc-4cd1-9a53-a15b42800de1', 'type': 'global_system_config', 'uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500'} 09/01/2025 01:57:34.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:35.876 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:35.878 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:35.879 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:35.879 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:35.879 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:35.880 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:35.880 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:35.881 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:35.881 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:35.881 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:35.881 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:35.881 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:35.882 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:35.882 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:35.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:36.941 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:37.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:38.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:39.880 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:39.882 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:39.882 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:39.883 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:39.883 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:39.883 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:39.883 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:39.884 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:39.884 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:39.884 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:39.884 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:39.884 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:39.885 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:39.885 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:39.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:40.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:41.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:42.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:43.883 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:43.885 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:43.885 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:43.886 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:43.886 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:43.886 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:43.887 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:43.887 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:43.888 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:43.888 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:43.888 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:43.889 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:43.889 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:43.889 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:43.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:44.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:45.957 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:46.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:47.886 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:47.889 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:47.890 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:47.890 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:47.891 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:47.891 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:47.891 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:47.893 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:47.893 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:47.893 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:47.894 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:47.894 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:47.895 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:47.895 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:47.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:48.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:50.910 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:51.890 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:51.893 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:51.893 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:51.894 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:51.894 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:51.895 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:51.895 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:51.896 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:51.896 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:51.897 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:51.897 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:51.897 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:51.898 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:51.898 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:51.911 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:52.913 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:53.915 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:54.916 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:55.894 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:55.897 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:55.897 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:55.897 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:55.898 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:55.898 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:55.898 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:55.899 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:55.899 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:55.900 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:55.900 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:55.900 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:55.900 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:55.900 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:55.918 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:56.919 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:57.920 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:58.922 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:59.898 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:59.900 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:59.900 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:59.900 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:59.901 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:59.901 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:59.901 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:59.902 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:59.902 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:59.902 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:59.902 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:59.903 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:59.903 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:59.903 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:59.923 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:00.924 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:01.925 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:02.927 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:03.901 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:03.903 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:03.903 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:03.904 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:03.904 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:03.904 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:03.904 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:03.905 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:03.906 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:03.906 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:03.906 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:03.906 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:03.906 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:03.907 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:03.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:04.931 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:05.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:06.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:07.904 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:07.907 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:07.907 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:07.908 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:07.908 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:07.908 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:07.909 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:07.910 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:07.910 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:07.910 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:07.910 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:07.911 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:07.911 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:07.911 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:07.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:08.937 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:09.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:10.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:11.908 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:11.910 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:11.910 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:11.911 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:11.911 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:11.911 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:11.911 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:11.912 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:11.913 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:11.913 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:11.913 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:11.913 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:11.914 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:11.914 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:11.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:12.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:13.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:14.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:15.910 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:15.913 7f75db8012c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:15.913 7f75db8012c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:15.913 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:15.914 7f75db8012c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:15.914 7f75db8012c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:15.914 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:15.915 7f75db8012c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:15.915 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:15.916 7f75db8012c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:15.916 7f75db8012c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:15.916 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:15.916 7f75db8012c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:15.916 7f75db8012c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:15.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:16.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:17.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:18.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:19.914 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:19.916 7f75db8012c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:19.919 7f75db8012c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:19.919 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:19.920 7f75db8012c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:19.920 7f75db8012c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:19.920 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:19.921 7f75db8012c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:19.921 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:19.922 7f75db8012c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:19.922 7f75db8012c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:19.922 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:19.923 7f75db8012c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:19.923 7f75db8012c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.911 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:21.913 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:22.915 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:23.916 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:23.917 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:23.918 7f75db8012c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:23.918 7f75db8012c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:23.918 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:23.918 7f75db8012c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:23.918 7f75db8012c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:23.918 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:23.919 7f75db8012c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:23.919 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:23.919 7f75db8012c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:23.919 7f75db8012c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:23.919 7f75db8012c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:23.919 7f75db8012c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:23.919 7f75db8012c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:24.918 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:25.919 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:26.921 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:27.918 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:27.920 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:27.921 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:27.921 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:27.921 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:27.922 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:27.922 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:27.923 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:27.923 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:27.923 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:27.924 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:27.924 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:27.924 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:27.924 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:27.925 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:28.927 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:29.929 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:30.930 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:31.922 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:31.925 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:31.925 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:31.925 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:31.926 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:31.926 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:31.926 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:31.927 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:31.927 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:31.927 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:31.927 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:31.928 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:31.928 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:31.928 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:31.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:32.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:33.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:34.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:35.927 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:35.929 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:35.929 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:35.929 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:35.930 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:35.930 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:35.930 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:35.931 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:35.931 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:35.932 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:35.932 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:35.932 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:35.932 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:35.933 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:35.939 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:36.941 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:37.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:38.945 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:39.930 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:39.934 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:39.934 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:39.934 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:39.935 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:39.935 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:39.935 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:39.936 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:39.937 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:39.937 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:39.937 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:39.938 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:39.938 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:39.938 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:39.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:40.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:41.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:42.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:43.935 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:43.941 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:43.941 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:43.941 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:43.942 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:43.942 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:43.942 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:43.942 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:43.943 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:43.943 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:43.943 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:43.943 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:43.943 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:43.943 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:43.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:44.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:45.957 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:46.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:47.941 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:47.944 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:47.945 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:47.945 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:47.946 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:47.946 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:47.946 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:47.947 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:47.947 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:47.948 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:47.948 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:47.949 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:47.949 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:47.949 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:47.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:48.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:50.912 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:51.914 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:51.945 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:51.947 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:51.948 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:51.948 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:51.948 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:51.949 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:51.949 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:51.949 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:51.950 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:51.950 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:51.950 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:51.950 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:51.951 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:51.951 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:52.916 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:53.919 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:54.922 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:55.924 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:55.948 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:55.952 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:55.953 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:55.953 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:55.954 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:55.954 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:55.954 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:55.955 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:55.955 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:55.956 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:55.956 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:55.957 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:55.957 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:55.957 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:56.874 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_routers in resource routing-instance: 'bgp_routers' 09/01/2025 01:58:56.875 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_refs in resource routing-instance: 'routing_instance_refs' 09/01/2025 01:58:56.875 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_target_refs in resource routing-instance: 'route_target_refs' 09/01/2025 01:58:56.875 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource routing-instance: 'tag_refs' 09/01/2025 01:58:56.875 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource routing-instance: 'virtual_machine_interface_back_refs' 09/01/2025 01:58:56.876 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_aggregate_back_refs in resource routing-instance: 'route_aggregate_back_refs' 09/01/2025 01:58:56.876 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_back_refs in resource routing-instance: 'routing_policy_back_refs' 09/01/2025 01:58:56.876 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_back_refs in resource routing-instance: 'routing_instance_back_refs' 09/01/2025 01:58:56.925 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3993-1'] not found in DB: Unknown id: bgp_router default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-ansible-os-3993-1 09/01/2025 01:58:56.926 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:56.928 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = bgp_router identifier_name = default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-ansible-os-3993-1 url = http://10.0.0.56:8082/bgp-routers operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.56 body = {"bgp-router": {"uuid": null, "fq_name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3993-1"], "parent_type": "routing-instance", "bgp_router_parameters": {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.14", "address": "10.20.0.14", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3993-1"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:58:56.953 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3993-1'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.14', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.14', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3993-1', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3993-1'], 'id_perms': {'created': '2025-09-01T01:58:56.941903', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:56.941903', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10796065273562621136, 'uuid_mslong': 17667355873221429929}}, 'parent_type': 'routing-instance', 'parent_uuid': '8105519c-daec-49ab-a7a7-a352adaaccd7', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f52f0e8f-9fc6-4aa9-95d3-542a92a164d0'}, 'oper': 'CREATE', 'request-id': 'req-fbe82a3d-e0cd-4f3c-a5ca-f9701286b080', 'type': 'bgp_router', 'uuid': 'f52f0e8f-9fc6-4aa9-95d3-542a92a164d0'} 09/01/2025 01:58:56.954 7f75daccbc20 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/01/2025 01:58:56.954 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:56.954 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.14", "address": "10.20.0.14", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-ansible-os-3993-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3993-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17667355873221429929, "uuid_lslong": 10796065273562621136}, "enable": true, "created": "2025-09-01T01:58:56.941903", "last_modified": "2025-09-01T01:58:56.941903", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "8105519c-daec-49ab-a7a7-a352adaaccd7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "f52f0e8f-9fc6-4aa9-95d3-542a92a164d0" } deleted = False >> 09/01/2025 01:58:56.954 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:57.002 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zone_refs in resource bgp-router: 'control_node_zone_refs' 09/01/2025 01:58:57.003 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource bgp-router: 'sub_cluster_refs' 09/01/2025 01:58:57.003 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource bgp-router: 'bgp_router_refs' 09/01/2025 01:58:57.003 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource bgp-router: 'tag_refs' 09/01/2025 01:58:57.003 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_system_config_back_refs in resource bgp-router: 'global_system_config_back_refs' 09/01/2025 01:58:57.003 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource bgp-router: 'physical_router_back_refs' 09/01/2025 01:58:57.004 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource bgp-router: 'virtual_machine_interface_back_refs' 09/01/2025 01:58:57.004 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_as_a_service_back_refs in resource bgp-router: 'bgp_as_a_service_back_refs' 09/01/2025 01:58:57.004 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_back_refs in resource bgp-router: 'bgp_router_back_refs' 09/01/2025 01:58:57.106 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:58:57.106 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:58:57.106 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:58:57.106 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:58:57.106 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:58:57.106 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:58:57.106 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:58:57.106 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:58:57.106 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:58:57.106 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:58:57.107 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:58:57.107 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:58:57.107 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:58:57.107 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:58:57.107 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:58:57.107 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:58:57.107 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:58:57.107 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:58:57.107 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:58:57.107 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:58:57.107 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:58:57.107 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:58:57.107 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:58:57.107 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:58:57.108 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:58:57.108 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:58:57.108 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:58:57.108 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:58:57.108 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:58:57.108 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:58:57.108 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:58:57.108 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:58:57.108 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:58:57.108 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:58:57.108 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:58:57.161 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 7eedf3ea-84b7-4623-96cf-7fecf55ca500 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.56:8082/ref-relax-for-delete operation = ref-relax-for-delete useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.56 body = {"uuid": "7eedf3ea-84b7-4623-96cf-7fecf55ca500", "ref-uuid": "f52f0e8f-9fc6-4aa9-95d3-542a92a164d0"} domain = default-domain project = admin user = admin >> 09/01/2025 01:58:57.210 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 7eedf3ea-84b7-4623-96cf-7fecf55ca500 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.56:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.56 body = {"type": "global-system-config", "uuid": "7eedf3ea-84b7-4623-96cf-7fecf55ca500", "ref-type": "bgp-router", "ref-uuid": "f52f0e8f-9fc6-4aa9-95d3-542a92a164d0", "ref-fq-name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3993-1"], "operation": "ADD", "attr": null} domain = default-domain project = admin user = admin >> 09/01/2025 01:58:57.217 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-0e8421c4-a285-47b4-9497-f3a607824657', 'type': 'global_system_config', 'uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500'} 09/01/2025 01:58:57.927 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:58.929 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:59.931 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:59.953 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:59.955 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:59.955 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:59.955 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:59.956 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:59.956 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:59.956 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:59.957 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:59.957 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:59.957 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:59.958 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:59.958 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:59.958 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:59.958 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:00.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:01.935 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:02.937 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:03.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:03.955 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:03.957 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:03.958 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:03.958 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:03.959 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:03.959 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:03.959 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:03.960 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:03.960 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:03.960 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:03.961 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:03.961 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:03.961 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:03.961 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:04.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:05.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:06.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:07.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:07.958 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:07.961 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:07.961 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:07.961 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:07.962 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:07.962 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:07.962 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:07.963 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:07.963 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:07.963 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:07.964 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:07.964 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:07.964 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:07.964 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:08.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:09.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:10.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:11.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:11.961 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:11.962 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:11.963 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:11.963 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:11.963 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:11.963 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:11.963 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:11.964 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:11.964 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:11.964 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:11.964 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:11.965 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:11.965 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:11.965 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:12.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:13.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:14.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:15.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:15.963 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:15.964 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:15.965 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:15.965 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:15.965 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:15.965 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:15.965 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:15.965 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:15.966 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:15.966 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:15.966 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:15.966 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:15.966 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:15.966 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:16.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:17.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:18.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:19.966 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:19.968 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:19.968 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:19.968 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:19.969 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:19.969 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:19.969 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:19.970 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:19.970 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.971 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:19.971 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.971 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.971 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:19.972 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:20.914 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:21.916 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:22.918 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:23.920 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:23.968 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:23.970 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:23.971 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:23.971 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:23.971 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:23.972 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:23.972 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:23.973 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:23.973 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:23.973 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:23.973 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:23.974 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:23.974 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:23.974 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:24.922 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:25.923 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:26.930 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:27.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:27.971 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:27.972 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:27.972 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:27.972 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:27.973 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:27.973 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:27.973 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:27.973 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:27.973 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:27.974 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:27.974 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:27.974 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:27.974 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:27.974 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:28.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:29.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:30.939 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:31.941 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:31.973 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:31.974 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:31.974 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:31.974 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:31.975 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:31.975 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:31.975 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:31.975 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:31.975 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:31.975 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:31.975 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:31.976 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:31.976 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:31.976 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:32.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:33.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:34.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:35.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:35.974 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:35.977 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:35.977 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:35.977 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:35.978 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:35.978 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:35.978 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:35.979 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:35.979 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:35.980 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:35.980 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:35.980 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:35.980 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:35.981 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:36.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:37.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:38.957 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:39.120 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:59:39.120 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:59:39.120 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:59:39.120 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:59:39.120 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:59:39.121 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:59:39.121 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:59:39.121 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:59:39.121 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:59:39.121 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:59:39.121 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:59:39.121 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:59:39.121 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:59:39.121 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:59:39.121 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:59:39.121 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:59:39.121 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:59:39.121 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:59:39.121 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:59:39.122 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:59:39.122 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:59:39.122 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:59:39.122 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:59:39.122 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:59:39.122 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:59:39.122 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:59:39.122 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:59:39.122 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:59:39.122 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:59:39.122 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:59:39.122 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:59:39.122 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:59:39.122 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:59:39.123 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:59:39.218 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'] not found in DB: Unknown id: database_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1. 09/01/2025 01:59:39.221 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = database_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1. url = http://10.0.0.56:8082/database-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_database_node.py remote_ip = 10.0.0.56 body = {"database-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1."], "parent_type": "global-system-config", "database_node_ip_address": "10.0.0.56", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3993-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:59:39.234 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'], 'obj_dict': {'database_node_ip_address': '10.0.0.56', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3993-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'], 'id_perms': {'created': '2025-09-01T01:59:39.230073', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:39.230073', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11940803800317669233, 'uuid_mslong': 12965038512893676413}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b3ed11d5-f449-4b7d-a5b6-41ca9291af71'}, 'oper': 'CREATE', 'request-id': 'req-f6b597f5-acb0-41b3-ae83-48a287481d21', 'type': 'database_node', 'uuid': 'b3ed11d5-f449-4b7d-a5b6-41ca9291af71'} 09/01/2025 01:59:39.234 7f75daccbc20 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:59:39.235 7f75da71b360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:39.235 7f75da71b360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. elements = { database_node_ip_address : "10.0.0.56"display_name : "cn-jenkins-deploy-platform-ansible-os-3993-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12965038512893676413, "uuid_lslong": 11940803800317669233}, "enable": true, "created": "2025-09-01T01:59:39.230073", "last_modified": "2025-09-01T01:59:39.230073", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "b3ed11d5-f449-4b7d-a5b6-41ca9291af71" } deleted = False >> 09/01/2025 01:59:39.235 7f75da71b360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:39.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:39.977 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:39.978 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:39.979 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:39.979 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:39.979 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:39.979 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:39.979 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:39.980 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:39.980 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:39.980 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:39.980 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:39.980 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:39.980 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:39.981 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:40.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:41.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:42.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:43.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:43.979 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:43.981 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:43.981 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:43.982 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:43.982 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:43.983 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:43.983 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:43.984 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:43.984 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:43.984 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:43.985 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:43.985 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:43.985 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:43.986 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:44.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:45.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:46.973 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:47.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:47.982 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:47.985 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:47.985 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:47.985 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:47.986 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:47.986 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:47.986 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:47.987 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:47.987 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:47.988 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:47.988 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:47.988 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:47.988 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:47.989 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:48.977 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:50.914 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:51.917 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:51.986 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:51.988 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:51.988 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:51.989 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:51.989 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:51.989 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:51.989 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:51.990 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:51.990 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:51.991 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:51.991 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:51.991 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:51.991 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:51.992 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:52.919 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:53.921 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:54.924 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:55.926 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:55.989 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:55.992 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:55.992 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:55.993 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:55.993 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:55.993 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:55.993 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:55.994 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:55.994 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:55.995 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:55.995 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:55.995 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:55.995 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:55.996 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:56.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:57.931 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:58.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:59.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:59.993 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:59.995 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:59.995 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:59.996 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:59.996 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:59.996 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:59.996 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:59.997 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:59.997 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:59.998 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:59.998 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:59.998 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:59.998 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:59.999 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:00.935 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:01.937 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:02.939 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:03.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:03.996 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:03.998 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:03.998 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:03.998 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:03.999 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:03.999 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:03.999 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:04.000 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:04.000 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:04.000 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:04.001 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:04.001 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:04.001 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:04.001 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:04.945 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:05.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:06.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:07.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:07.999 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:08.001 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:08.001 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:08.001 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:08.002 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:08.002 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:08.002 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:08.003 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:08.004 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:08.004 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:08.004 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:08.004 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:08.005 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:08.005 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:08.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:09.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:10.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:11.981 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:12.002 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:12.004 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:12.005 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:12.005 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:12.006 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:12.006 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:12.006 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:12.007 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:12.007 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:12.008 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:12.008 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:12.008 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:12.008 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:12.008 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:12.983 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:13.984 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:14.986 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:15.988 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:16.005 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:16.006 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:16.007 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:16.007 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:16.007 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:16.007 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:16.007 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:16.007 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:16.007 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:16.008 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:16.008 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:16.008 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:16.008 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:16.008 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:16.991 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:17.992 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:18.994 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:20.008 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:20.010 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:20.010 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:20.011 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:20.011 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:20.011 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:20.011 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:20.012 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:20.012 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:20.013 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:20.013 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:20.013 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:20.013 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:20.014 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:20.915 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:21.917 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:22.919 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:23.921 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:24.010 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:24.012 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:24.013 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:24.013 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:24.014 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:24.014 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:24.014 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:24.015 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:24.015 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:24.016 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:24.016 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:24.016 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:24.016 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:24.017 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:24.923 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:25.924 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:26.926 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:27.113 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 02:00:27.113 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 02:00:27.113 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 02:00:27.113 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 02:00:27.113 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 02:00:27.113 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 02:00:27.113 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 02:00:27.114 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 02:00:27.114 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 02:00:27.114 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 02:00:27.114 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 02:00:27.114 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 02:00:27.114 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 02:00:27.114 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 02:00:27.114 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 02:00:27.114 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 02:00:27.114 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 02:00:27.114 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 02:00:27.114 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 02:00:27.115 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 02:00:27.115 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 02:00:27.115 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 02:00:27.115 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 02:00:27.115 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 02:00:27.115 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 02:00:27.115 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 02:00:27.115 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 02:00:27.115 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 02:00:27.115 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 02:00:27.115 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 02:00:27.115 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 02:00:27.116 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 02:00:27.116 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 02:00:27.116 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 02:00:27.216 7f75db801180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'] not found in DB: Unknown id: analytics_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1. 09/01/2025 02:00:27.220 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1. url = http://10.0.0.56:8082/analytics-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_analytics_node.py remote_ip = 10.0.0.56 body = {"analytics-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1."], "parent_type": "global-system-config", "analytics_node_ip_address": "10.0.0.56", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3993-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 02:00:27.245 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.56', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3993-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'], 'id_perms': {'created': '2025-09-01T02:00:27.235470', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:00:27.235470', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12323209430845218635, 'uuid_mslong': 3889598271637308046}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '35faa105-23ea-428e-ab04-d5afaa45df4b'}, 'oper': 'CREATE', 'request-id': 'req-39a1b43c-ee87-49bf-b77f-cc837784a2a9', 'type': 'analytics_node', 'uuid': '35faa105-23ea-428e-ab04-d5afaa45df4b'} 09/01/2025 02:00:27.246 7f75daccbc20 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/01/2025 02:00:27.247 7f75da7db400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:27.248 7f75da7db400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. elements = { analytics_node_ip_address : "10.0.0.56"display_name : "cn-jenkins-deploy-platform-ansible-os-3993-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3889598271637308046, "uuid_lslong": 12323209430845218635}, "enable": true, "created": "2025-09-01T02:00:27.235470", "last_modified": "2025-09-01T02:00:27.235470", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "35faa105-23ea-428e-ab04-d5afaa45df4b" } deleted = False >> 09/01/2025 02:00:27.248 7f75da7db400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:27.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:28.014 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:28.016 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:28.016 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:28.016 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:28.017 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:28.017 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:28.017 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:28.018 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:28.018 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:28.019 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:28.019 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:28.019 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:28.020 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:28.020 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:28.930 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:29.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:30.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:31.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:32.016 7f75db801180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:32.018 7f75db801180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:32.018 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:32.018 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:32.018 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:32.018 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:32.019 7f75db801180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:32.019 7f75db801180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:32.019 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:32.019 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:32.019 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:32.019 7f75db801180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:32.020 7f75db801180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:32.020 7f75db801180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:32.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:33.939 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:34.941 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:35.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:36.019 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:36.020 7f75da7db2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:36.020 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:36.020 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:36.021 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:36.021 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:36.021 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:36.021 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:36.021 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:36.022 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:36.022 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:36.022 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:36.022 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:36.022 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:36.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:37.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:38.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:39.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:40.021 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:40.023 7f75da7db2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:40.023 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:40.024 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:40.024 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:40.024 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:40.025 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:40.026 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:40.026 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:40.026 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:40.027 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:40.027 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:40.027 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:40.028 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:40.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:41.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:42.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:43.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:44.024 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:44.026 7f75da7db2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:44.026 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:44.026 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:44.027 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:44.027 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:44.027 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:44.028 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:44.028 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:44.029 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:44.029 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:44.029 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:44.029 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:44.030 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:44.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:45.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:46.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:47.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:48.026 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:48.029 7f75da7db2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:48.029 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:48.029 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:48.030 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:48.030 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:48.030 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:48.031 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:48.031 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:48.031 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:48.032 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:48.032 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:48.032 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:48.032 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:48.570 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 02:00:48.571 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 02:00:48.571 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 02:00:48.572 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 02:00:48.572 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 02:00:48.572 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 02:00:48.573 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 02:00:48.573 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 02:00:48.573 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 02:00:48.574 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 02:00:48.574 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 02:00:48.574 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 02:00:48.574 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 02:00:48.575 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 02:00:48.575 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 02:00:48.575 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 02:00:48.576 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 02:00:48.576 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 02:00:48.576 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 02:00:48.576 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 02:00:48.577 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 02:00:48.577 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 02:00:48.577 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 02:00:48.578 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 02:00:48.578 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 02:00:48.578 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 02:00:48.578 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 02:00:48.579 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 02:00:48.579 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 02:00:48.579 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 02:00:48.579 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 02:00:48.580 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 02:00:48.580 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 02:00:48.580 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 02:00:48.684 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'] not found in DB: Unknown id: analytics_alarm_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1. 09/01/2025 02:00:48.692 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_alarm_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1. url = http://10.0.0.56:8082/analytics-alarm-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_analytics_alarm_node.py remote_ip = 10.0.0.56 body = {"analytics-alarm-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1."], "parent_type": "global-system-config", "analytics_alarm_node_ip_address": "10.0.0.56", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3993-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 02:00:48.723 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.56', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3993-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'], 'id_perms': {'created': '2025-09-01T02:00:48.714306', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:00:48.714306', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13408469371240238503, 'uuid_mslong': 10676906133802273316}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '942bfd8f-5d79-4a24-ba14-73da1e7acda7'}, 'oper': 'CREATE', 'request-id': 'req-d9fddf6e-4f72-4e37-b794-27c0e82736cc', 'type': 'analytics_alarm_node', 'uuid': '942bfd8f-5d79-4a24-ba14-73da1e7acda7'} 09/01/2025 02:00:48.724 7f75daccbc20 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/01/2025 02:00:48.724 7f75da71b680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:48.725 7f75da71b680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. elements = { analytics_alarm_node_ip_address : "10.0.0.56"display_name : "cn-jenkins-deploy-platform-ansible-os-3993-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10676906133802273316, "uuid_lslong": 13408469371240238503}, "enable": true, "created": "2025-09-01T02:00:48.714306", "last_modified": "2025-09-01T02:00:48.714306", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "942bfd8f-5d79-4a24-ba14-73da1e7acda7" } deleted = False >> 09/01/2025 02:00:48.725 7f75da71b680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:48.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:50.914 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:51.916 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:52.030 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:52.032 7f75da7db2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:52.032 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:52.033 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:52.033 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:52.033 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:52.033 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:52.034 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:52.034 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:52.035 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:52.035 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:52.035 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:52.035 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:52.035 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:52.918 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:53.920 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:54.922 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:55.923 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:56.032 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:56.034 7f75da7db2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:56.034 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:56.034 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:56.034 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:56.034 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:56.035 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:56.035 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:56.035 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:56.035 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:56.035 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:56.036 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:56.036 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:56.036 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:56.924 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:57.925 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:58.926 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:59.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:00.034 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:00.037 7f75da7db2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:00.037 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:00.037 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:00.038 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:00.038 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:00.038 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:00.039 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:00.039 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:00.040 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:00.040 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:00.040 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:00.040 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:00.041 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:00.930 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:01.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:02.939 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:03.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:04.037 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:04.041 7f75da7db2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:04.042 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:04.042 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:04.042 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:04.042 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:04.043 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:04.043 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:04.043 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:04.044 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:04.044 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:04.044 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:04.044 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:04.044 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:04.848 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 02:01:04.848 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 02:01:04.848 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 02:01:04.849 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 02:01:04.849 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 02:01:04.849 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 02:01:04.849 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 02:01:04.849 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 02:01:04.850 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 02:01:04.850 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 02:01:04.850 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 02:01:04.850 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 02:01:04.850 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 02:01:04.850 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 02:01:04.851 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 02:01:04.851 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 02:01:04.851 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 02:01:04.851 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 02:01:04.851 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 02:01:04.851 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 02:01:04.852 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 02:01:04.852 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 02:01:04.852 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 02:01:04.852 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 02:01:04.852 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 02:01:04.853 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 02:01:04.853 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 02:01:04.853 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 02:01:04.853 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 02:01:04.853 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 02:01:04.854 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 02:01:04.854 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 02:01:04.854 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 02:01:04.854 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 02:01:04.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:04.960 7f75da7db2c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'] not found in DB: Unknown id: analytics_snmp_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1. 09/01/2025 02:01:04.969 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_snmp_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1. url = http://10.0.0.56:8082/analytics-snmp-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_analytics_snmp_node.py remote_ip = 10.0.0.56 body = {"analytics-snmp-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1."], "parent_type": "global-system-config", "analytics_snmp_node_ip_address": "10.0.0.56", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3993-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 02:01:05.000 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.56', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3993-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'], 'id_perms': {'created': '2025-09-01T02:01:04.990043', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:01:04.990043', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11043142725590816080, 'uuid_mslong': 5586765827790422298}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4d882deb-a0bd-411a-9941-1fccc2c79550'}, 'oper': 'CREATE', 'request-id': 'req-05faaaa3-e349-456f-a99e-6905adfb8346', 'type': 'analytics_snmp_node', 'uuid': '4d882deb-a0bd-411a-9941-1fccc2c79550'} 09/01/2025 02:01:05.001 7f75daccbc20 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 09/01/2025 02:01:05.001 7f75da71b860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.002 7f75da71b860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. elements = { analytics_snmp_node_ip_address : "10.0.0.56"display_name : "cn-jenkins-deploy-platform-ansible-os-3993-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 5586765827790422298, "uuid_lslong": 11043142725590816080}, "enable": true, "created": "2025-09-01T02:01:04.990043", "last_modified": "2025-09-01T02:01:04.990043", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "4d882deb-a0bd-411a-9941-1fccc2c79550" } deleted = False >> 09/01/2025 02:01:05.002 7f75da71b860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:06.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:07.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:08.042 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:08.044 7f75da7db2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:08.044 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:08.045 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:08.045 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:08.045 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:08.045 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:08.046 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:08.046 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:08.046 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:08.047 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:08.047 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:08.047 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:08.047 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:08.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:09.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:10.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:11.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:12.045 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:12.046 7f75da7db2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:12.047 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:12.047 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:12.047 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:12.047 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:12.047 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:12.047 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:12.047 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:12.048 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:12.048 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:12.048 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:12.048 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:12.048 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:12.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:13.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:14.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:15.965 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:16.047 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:16.048 7f75da7db2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:16.048 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:16.048 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:16.049 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:16.049 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:16.049 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:16.049 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:16.049 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:16.049 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:16.049 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:16.049 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:16.049 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:16.049 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:16.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:17.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:18.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:20.049 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:20.052 7f75da7db2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:20.052 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:20.053 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:20.053 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:20.053 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:20.054 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:20.055 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:20.055 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:20.055 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:20.056 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:20.056 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:20.056 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:20.057 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:20.917 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:21.919 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:22.921 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:23.926 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:24.052 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:24.055 7f75da7db2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:24.055 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:24.055 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:24.055 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:24.056 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:24.056 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:24.057 7f75da7db2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:24.057 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:24.057 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:24.057 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:24.058 7f75da7db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:24.058 7f75da7db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:24.058 7f75da7db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:24.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:25.929 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:26.930 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:27.931 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:28.055 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:28.058 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:28.059 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:28.059 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:28.059 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:28.059 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:28.060 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:28.060 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:28.061 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:28.061 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:28.061 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:28.061 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:28.062 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:28.062 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:28.933 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:29.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:30.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:31.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:32.059 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:32.061 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:32.062 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:32.062 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:32.062 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:32.063 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:32.063 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:32.064 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:32.064 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:32.064 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:32.064 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:32.064 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:32.065 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:32.065 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:32.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:33.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:34.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:35.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:36.062 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:36.065 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:36.065 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:36.066 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:36.066 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:36.066 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:36.066 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:36.067 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:36.067 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:36.068 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:36.068 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:36.068 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:36.069 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:36.069 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:36.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:37.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:38.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:39.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:40.065 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:40.067 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:40.067 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:40.067 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:40.067 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:40.067 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:40.067 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:40.068 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:40.068 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:40.068 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:40.068 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:40.068 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:40.068 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:40.068 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:40.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:41.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:42.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:43.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:44.067 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:44.068 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:44.068 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:44.068 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:44.069 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:44.069 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:44.069 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:44.069 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:44.069 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:44.070 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:44.070 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:44.070 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:44.070 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:44.070 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:44.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:45.965 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:46.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:47.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:48.069 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:48.071 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:48.071 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:48.071 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:48.072 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:48.072 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:48.072 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:48.073 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:48.073 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:48.074 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:48.074 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:48.074 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:48.074 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:48.075 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:48.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:50.917 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:51.919 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:52.071 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:52.072 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:52.072 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:52.072 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:52.073 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:52.073 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:52.073 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:52.073 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:52.073 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:52.073 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:52.073 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:52.073 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:52.073 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:52.074 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:52.922 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:53.925 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:54.926 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:55.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:56.073 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:56.075 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:56.076 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:56.076 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:56.076 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:56.077 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:56.077 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:56.078 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:56.078 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:56.078 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:56.078 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:56.079 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:56.079 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:56.079 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:56.931 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:57.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:58.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:59.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:00.076 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:00.078 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:00.078 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:00.079 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:00.079 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:00.079 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:00.079 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:00.080 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:00.080 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:00.080 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:00.080 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:00.081 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:00.081 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:00.081 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:00.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:01.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:02.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:03.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:04.079 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:04.081 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:04.082 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:04.082 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:04.083 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:04.083 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:04.083 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:04.084 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:04.084 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:04.085 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:04.085 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:04.085 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:04.086 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:04.086 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:04.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:05.957 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:06.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:07.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:08.082 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:08.084 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:08.085 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:08.085 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:08.085 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:08.086 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:08.086 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:08.087 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:08.087 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:08.087 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:08.087 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:08.087 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:08.088 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:08.088 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:08.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:09.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:10.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:11.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:12.085 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:12.087 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:12.088 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:12.088 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:12.088 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:12.089 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:12.089 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:12.089 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:12.090 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:12.090 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:12.090 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:12.090 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:12.090 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:12.091 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:12.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:13.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:14.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:15.973 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:16.089 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:16.091 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:16.092 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:16.092 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:16.093 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:16.093 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:16.093 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:16.095 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:16.095 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:16.096 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:16.096 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:16.096 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:16.097 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:16.097 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:16.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:17.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:18.979 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:20.092 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:20.094 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:20.094 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:20.095 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:20.095 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:20.095 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:20.095 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:20.096 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:20.096 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:20.096 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:20.097 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:20.097 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:20.097 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:20.097 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:20.918 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:21.920 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:22.922 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:23.924 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:24.095 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:24.098 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:24.098 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:24.099 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:24.099 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:24.100 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:24.100 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:24.101 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:24.101 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:24.102 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:24.102 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:24.103 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:24.103 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:24.103 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:24.925 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:25.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:26.930 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:27.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:28.099 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:28.101 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:28.101 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:28.102 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:28.102 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:28.102 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:28.102 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:28.103 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:28.104 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:28.104 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:28.104 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:28.104 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:28.105 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:28.105 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:28.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:29.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:30.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:31.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:32.102 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:32.104 7f75da7db4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:32.105 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:32.105 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:32.105 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:32.106 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:32.106 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:32.107 7f75da7db4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:32.107 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:32.107 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:32.107 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:32.108 7f75da7db4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:32.108 7f75da7db4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:32.108 7f75da7db4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:32.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:33.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:34.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:35.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:36.105 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:36.107 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:36.107 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:36.108 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:36.108 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:36.108 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:36.108 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:36.109 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:36.109 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:36.110 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:36.110 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:36.110 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:36.110 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:36.111 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:36.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:37.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:38.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:39.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:40.108 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:40.110 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:40.111 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:40.111 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:40.111 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:40.112 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:40.112 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:40.113 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:40.113 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:40.113 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:40.113 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:40.114 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:40.114 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:40.114 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:40.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:41.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:42.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:43.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:44.111 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:44.114 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:44.114 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:44.114 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:44.115 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:44.115 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:44.115 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:44.116 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:44.116 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:44.117 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:44.117 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:44.117 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:44.117 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:44.117 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:44.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:45.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:46.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:47.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:48.114 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:48.117 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:48.117 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:48.118 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:48.118 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:48.118 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:48.118 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:48.119 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:48.120 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:48.120 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:48.120 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:48.120 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:48.121 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:48.121 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:48.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:50.920 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:51.921 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:52.118 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:52.120 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:52.120 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:52.121 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:52.121 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:52.121 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:52.121 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:52.121 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:52.121 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:52.121 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:52.121 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:52.121 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:52.121 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:52.121 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:52.924 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:53.926 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:54.927 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:55.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:56.121 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:56.123 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:56.124 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:56.124 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:56.125 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:56.125 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:56.125 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:56.126 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:56.126 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:56.127 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:56.127 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:56.127 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:56.127 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:56.128 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:56.930 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:57.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:58.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:59.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:00.124 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:00.126 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:00.126 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:00.126 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:00.126 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:00.126 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:00.127 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:00.127 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:00.127 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:00.127 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:00.128 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:00.128 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:00.128 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:00.128 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:00.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:01.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:02.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:03.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:04.127 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:04.129 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:04.129 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:04.129 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:04.130 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:04.130 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:04.130 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:04.131 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:04.131 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:04.131 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:04.132 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:04.132 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:04.132 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:04.132 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:04.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:05.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:06.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:07.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:08.130 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:08.132 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:08.132 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:08.132 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:08.133 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:08.133 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:08.133 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:08.134 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:08.134 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:08.134 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:08.135 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:08.135 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:08.135 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:08.135 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:08.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:09.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:10.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:11.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:12.132 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:12.134 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:12.135 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:12.135 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:12.135 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:12.136 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:12.136 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:12.137 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:12.137 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:12.137 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:12.137 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:12.138 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:12.138 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:12.138 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:12.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:13.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:14.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:15.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:16.135 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:16.137 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:16.138 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:16.138 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:16.139 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:16.139 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:16.139 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:16.140 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:16.141 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:16.141 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:16.142 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:16.142 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:16.142 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:16.143 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:16.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:17.973 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:18.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:20.139 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:20.141 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:20.141 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:20.142 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:20.142 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:20.142 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:20.143 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:20.143 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:20.144 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:20.144 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:20.144 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:20.144 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:20.145 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:20.145 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:20.920 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:21.922 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:22.924 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:23.926 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:24.142 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:24.144 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:24.145 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:24.145 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:24.145 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:24.146 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:24.146 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:24.147 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:24.147 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:24.147 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:24.147 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:24.148 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:24.148 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:24.148 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:24.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:25.930 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:26.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:27.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:28.145 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:28.147 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:28.148 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:28.148 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:28.148 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:28.149 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:28.149 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:28.149 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:28.150 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:28.150 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:28.150 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:28.150 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:28.151 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:28.151 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:28.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:29.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:30.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:31.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:32.149 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:32.150 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:32.150 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:32.150 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:32.150 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:32.151 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:32.151 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:32.151 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:32.151 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:32.151 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:32.152 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:32.152 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:32.152 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:32.152 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:32.945 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:33.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:34.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:35.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:36.151 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:36.153 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:36.154 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:36.154 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:36.154 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:36.155 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:36.155 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:36.156 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:36.156 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:36.156 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:36.156 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:36.157 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:36.157 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:36.157 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:36.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:37.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:38.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:39.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:40.154 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:40.156 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:40.157 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:40.157 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:40.158 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:40.158 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:40.158 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:40.159 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:40.159 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:40.159 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:40.159 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:40.160 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:40.160 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:40.160 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:40.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:41.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:42.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:43.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:44.157 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:44.160 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:44.160 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:44.160 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:44.161 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:44.161 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:44.161 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:44.162 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:44.162 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:44.163 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:44.163 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:44.163 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:44.163 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:44.163 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:44.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:45.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:46.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:47.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:48.161 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:48.163 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:48.163 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:48.163 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:48.163 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:48.164 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:48.164 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:48.165 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:48.165 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:48.166 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:48.166 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:48.166 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:48.166 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:48.166 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:48.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:50.921 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:51.923 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:52.163 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:52.164 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:52.165 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:52.165 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:52.165 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:52.165 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:52.165 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:52.165 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:52.165 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:52.165 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:52.166 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:52.166 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:52.166 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:52.166 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:52.924 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:53.926 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:54.927 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:55.929 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:56.006 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 02:03:56.007 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 02:03:56.007 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 02:03:56.008 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 02:03:56.008 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 02:03:56.009 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 02:03:56.009 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 02:03:56.010 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 02:03:56.010 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 02:03:56.011 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 02:03:56.011 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 02:03:56.012 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 02:03:56.012 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 02:03:56.013 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 02:03:56.013 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 02:03:56.014 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 02:03:56.014 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 02:03:56.014 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 02:03:56.015 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 02:03:56.015 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 02:03:56.016 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 02:03:56.016 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 02:03:56.016 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 02:03:56.017 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 02:03:56.017 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 02:03:56.018 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 02:03:56.018 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 02:03:56.018 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 02:03:56.019 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 02:03:56.019 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 02:03:56.020 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 02:03:56.020 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 02:03:56.021 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 02:03:56.021 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 02:03:56.125 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_routers in resource routing-instance: 'bgp_routers' 09/01/2025 02:03:56.125 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_refs in resource routing-instance: 'routing_instance_refs' 09/01/2025 02:03:56.126 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_target_refs in resource routing-instance: 'route_target_refs' 09/01/2025 02:03:56.126 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource routing-instance: 'tag_refs' 09/01/2025 02:03:56.126 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource routing-instance: 'virtual_machine_interface_back_refs' 09/01/2025 02:03:56.127 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_aggregate_back_refs in resource routing-instance: 'route_aggregate_back_refs' 09/01/2025 02:03:56.127 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_back_refs in resource routing-instance: 'routing_policy_back_refs' 09/01/2025 02:03:56.128 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_back_refs in resource routing-instance: 'routing_instance_back_refs' 09/01/2025 02:03:56.166 7f75da71b720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:56.168 7f75da71b720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:56.168 7f75da71b720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:56.168 7f75da71b720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:56.169 7f75da71b720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:56.169 7f75da71b720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:56.169 7f75da71b720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:56.170 7f75da71b720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:56.170 7f75da71b720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:56.170 7f75da71b720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:56.171 7f75da71b720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:56.171 7f75da71b720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:56.171 7f75da71b720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:56.171 7f75da71b720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:56.186 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 7eedf3ea-84b7-4623-96cf-7fecf55ca500 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.56:8082/global-system-config/7eedf3ea-84b7-4623-96cf-7fecf55ca500 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.56 body = {"global-system-config": {"uuid": "7eedf3ea-84b7-4623-96cf-7fecf55ca500", "ip_fabric_subnets": {"subnet": [{"ip_prefix": "10.20.0.0", "ip_prefix_len": 24}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 02:03:56.208 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-5d92ffbe-d671-4442-9a6a-cbeb361f2ec9', 'type': 'global_system_config', 'uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500'} 09/01/2025 02:03:56.308 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1'] not found in DB: Unknown id: virtual_router default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1 09/01/2025 02:03:56.316 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = virtual_router identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1 url = http://10.0.0.56:8082/virtual-routers operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.56 body = {"virtual-router": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1"], "parent_type": "global-system-config", "virtual_router_dpdk_enabled": false, "virtual_router_ip_address": "10.20.0.14", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3993-1"}} domain = default-domain project = admin user = admin >> 09/01/2025 02:03:56.348 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1'], 'obj_dict': {'display_name': 'cn-jenkins-deploy-platform-ansible-os-3993-1', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1'], 'id_perms': {'created': '2025-09-01T02:03:56.337011', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:03:56.337011', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13614653776856471694, 'uuid_mslong': 13705629618940693837}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'be342d85-dc08-4d4d-bcf0-f779fd75d88e', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.14'}, 'oper': 'CREATE', 'request-id': 'req-d72e219f-e9c4-4d0e-b3c6-3076bbe59a13', 'type': 'virtual_router', 'uuid': 'be342d85-dc08-4d4d-bcf0-f779fd75d88e'} 09/01/2025 02:03:56.349 7f75daccbc20 [contrail-api] [DEBUG]: Add uve in the [ObjectVRouter:ContrailConfigTrace] map 09/01/2025 02:03:56.351 7f75da71b680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:56.351 7f75da71b680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1 elements = { display_name : "cn-jenkins-deploy-platform-ansible-os-3993-1"fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 13705629618940693837, "uuid_lslong": 13614653776856471694}, "enable": true, "created": "2025-09-01T02:03:56.337011", "last_modified": "2025-09-01T02:03:56.337011", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "be342d85-dc08-4d4d-bcf0-f779fd75d88e"virtual_router_dpdk_enabled : falsevirtual_router_ip_address : "10.20.0.14" } deleted = False >> 09/01/2025 02:03:56.351 7f75da71b680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:56.455 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:03:56.456 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:03:56.456 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:03:56.457 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:03:56.457 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:03:56.457 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:03:56.457 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:03:56.458 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:03:56.621 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field access_control_lists in resource virtual-network: 'access_control_lists' 09/01/2025 02:03:56.621 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field floating_ip_pools in resource virtual-network: 'floating_ip_pools' 09/01/2025 02:03:56.621 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alias_ip_pools in resource virtual-network: 'alias_ip_pools' 09/01/2025 02:03:56.621 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instances in resource virtual-network: 'routing_instances' 09/01/2025 02:03:56.621 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bridge_domains in resource virtual-network: 'bridge_domains' 09/01/2025 02:03:56.622 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_object_refs in resource virtual-network: 'security_logging_object_refs' 09/01/2025 02:03:56.622 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_refs in resource virtual-network: 'qos_config_refs' 09/01/2025 02:03:56.622 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-network: 'network_ipam_refs' 09/01/2025 02:03:56.622 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_policy_refs in resource virtual-network: 'network_policy_refs' 09/01/2025 02:03:56.622 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_refs in resource virtual-network: 'virtual_network_refs' 09/01/2025 02:03:56.622 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_health_check_refs in resource virtual-network: 'service_health_check_refs' 09/01/2025 02:03:56.622 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_table_refs in resource virtual-network: 'route_table_refs' 09/01/2025 02:03:56.622 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field multicast_policy_refs in resource virtual-network: 'multicast_policy_refs' 09/01/2025 02:03:56.622 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgpvpn_refs in resource virtual-network: 'bgpvpn_refs' 09/01/2025 02:03:56.622 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_map_refs in resource virtual-network: 'intent_map_refs' 09/01/2025 02:03:56.623 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_refs in resource virtual-network: 'routing_policy_refs' 09/01/2025 02:03:56.623 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-network: 'tag_refs' 09/01/2025 02:03:56.623 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_back_refs in resource virtual-network: 'virtual_network_back_refs' 09/01/2025 02:03:56.623 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource virtual-network: 'virtual_machine_interface_back_refs' 09/01/2025 02:03:56.623 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-network: 'instance_ip_back_refs' 09/01/2025 02:03:56.623 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-network: 'physical_router_back_refs' 09/01/2025 02:03:56.623 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field port_tuple_back_refs in resource virtual-network: 'port_tuple_back_refs' 09/01/2025 02:03:56.623 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field logical_router_back_refs in resource virtual-network: 'logical_router_back_refs' 09/01/2025 02:03:56.623 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_node_back_refs in resource virtual-network: 'flow_node_back_refs' 09/01/2025 02:03:56.623 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field firewall_rule_back_refs in resource virtual-network: 'firewall_rule_back_refs' 09/01/2025 02:03:56.624 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnect_back_refs in resource virtual-network: 'data_center_interconnect_back_refs' 09/01/2025 02:03:56.624 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabric_back_refs in resource virtual-network: 'fabric_back_refs' 09/01/2025 02:03:56.624 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field host_based_service_back_refs in resource virtual-network: 'host_based_service_back_refs' 09/01/2025 02:03:56.684 7f75da7db0e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1', 'vhost0'] not found in DB: Unknown id: virtual_machine_interface default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1:vhost0 09/01/2025 02:03:56.691 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = virtual_machine_interface identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1:vhost0 url = http://10.0.0.56:8082/virtual-machine-interfaces operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.56 body = {"virtual-machine-interface": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1", "vhost0"], "parent_type": "virtual-router", "virtual_machine_interface_disable_policy": true, "virtual_network_refs": [{"to": ["default-domain", "default-project", "ip-fabric"], "uuid": "c39bc5a7-3ad4-482b-9be0-1adbb0ac1242"}], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "vhost0"}} domain = default-domain project = admin user = admin >> 09/01/2025 02:03:56.735 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:03:56.715607', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:03:56.715607', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10136671416836121550, 'uuid_mslong': 1509424230754177005}}, 'parent_type': 'virtual-router', 'parent_uuid': 'be342d85-dc08-4d4d-bcf0-f779fd75d88e', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '14f28d57-3178-4fed-8cac-b0f27d5f63ce', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:14:f2:8d:57:31']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'c39bc5a7-3ad4-482b-9be0-1adbb0ac1242'}]}, 'oper': 'CREATE', 'request-id': 'req-20622f07-cda6-43e3-8080-4a5fb623365b', 'type': 'virtual_machine_interface', 'uuid': '14f28d57-3178-4fed-8cac-b0f27d5f63ce'} 09/01/2025 02:03:56.736 7f75daccbc20 [contrail-api] [DEBUG]: Add uve in the [ObjectVMITable:ContrailConfigTrace] map 09/01/2025 02:03:56.738 7f75da71b5e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:56.738 7f75da71b5e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1509424230754177005, "uuid_lslong": 10136671416836121550}, "enable": true, "created": "2025-09-01T02:03:56.715607", "last_modified": "2025-09-01T02:03:56.715607", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "be342d85-dc08-4d4d-bcf0-f779fd75d88e"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "14f28d57-3178-4fed-8cac-b0f27d5f63ce"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:14:f2:8d:57:31"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "uuid": "c39bc5a7-3ad4-482b-9be0-1adbb0ac1242"}] } deleted = False >> 09/01/2025 02:03:56.738 7f75da71b5e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:56.745 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 14f28d57-3178-4fed-8cac-b0f27d5f63ce object_type = virtual_machine_interface identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1:vhost0 url = http://127.0.0.1/ref-update operation = ref-update domain = default-domain >> 09/01/2025 02:03:56.772 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-20622f07-cda6-43e3-8080-4a5fb623365b', 'type': 'virtual_machine_interface', 'uuid': '14f28d57-3178-4fed-8cac-b0f27d5f63ce'} 09/01/2025 02:03:56.783 7f75da71b680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:56.783 7f75da71b680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3993-1:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3993-1", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1509424230754177005, "uuid_lslong": 10136671416836121550}, "enable": true, "created": "2025-09-01T02:03:56.715607", "last_modified": "2025-09-01T02:03:56.752892", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "be342d85-dc08-4d4d-bcf0-f779fd75d88e"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}routing_instance_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "ip-fabric"], "attr": {"direction": "both", "vlan_tag": null, "src_mac": null, "dst_mac": null, "mpls_label": null, "service_chain_address": null, "ipv6_service_chain_address": null, "protocol": null}, "uuid": "231b437c-f8ca-4984-8584-b66c8b6969d7"}]uuid : "14f28d57-3178-4fed-8cac-b0f27d5f63ce"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:14:f2:8d:57:31"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "attr": null, "uuid": "c39bc5a7-3ad4-482b-9be0-1adbb0ac1242"}] } deleted = False >> 09/01/2025 02:03:56.784 7f75da71b680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:56.931 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:57.297 7f75daa595e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 699e2991-a098-40cb-81fd-a7e669d68ed7 object_type = access_control_list identifier_name = default-domain:default-project:ip-fabric:ip-fabric url = http://10.0.0.56:8082/access-control-list/699e2991-a098-40cb-81fd-a7e669d68ed7 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3993-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.56 body = {"access-control-list": {"uuid": "699e2991-a098-40cb-81fd-a7e669d68ed7", "access_control_list_entries": {"dynamic": false, "acl_rule": [{"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "deny", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000002", "direction": "<>"}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}]}, "access_control_list_hash": 7756030073107004076}} domain = default-domain project = admin user = admin >> 09/01/2025 02:03:57.307 7f75daccbc20 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-375f7284-d086-48a3-96ae-3f89b820eabc', 'type': 'access_control_list', 'uuid': '699e2991-a098-40cb-81fd-a7e669d68ed7'} 09/01/2025 02:03:57.933 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:58.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:59.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:00.168 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:00.169 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:00.169 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:00.170 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:00.170 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:00.170 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:00.170 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:00.170 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:00.170 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:00.170 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:00.170 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:00.170 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:00.171 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:00.171 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:00.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:01.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:02.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:03.945 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:04.170 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:04.171 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:04.172 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:04.172 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:04.172 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:04.172 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:04.172 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:04.173 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:04.173 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:04.173 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:04.173 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:04.173 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:04.173 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:04.174 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:04.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:05.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:06.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:07.791 7f75da7db0e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:04:07.799 7f75da7db0e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:04:07.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:08.172 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:08.175 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:08.175 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:08.175 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:08.175 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:08.176 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:08.176 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:08.176 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:08.177 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:08.177 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:08.177 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:08.177 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:08.177 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:08.178 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:08.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:09.957 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:10.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:11.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:12.175 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:12.178 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:12.178 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:12.178 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:12.179 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:12.179 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:12.179 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:12.180 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:12.180 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:12.181 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:12.181 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:12.181 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:12.182 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:12.182 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:12.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:13.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:14.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:15.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:16.179 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:16.180 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:16.181 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:16.181 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:16.181 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:16.181 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:16.181 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:16.182 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:16.182 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:16.182 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:16.182 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:16.182 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:16.183 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:16.183 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:16.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:17.973 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:18.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:20.182 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:20.186 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:20.186 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:20.186 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:20.187 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:20.187 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:20.187 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:20.188 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:20.188 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:20.189 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:20.189 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:20.189 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:20.189 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:20.190 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:20.707 7f75da7db0e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:04:20.714 7f75da7db0e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:04:20.923 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:21.924 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:22.926 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:23.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:24.187 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:24.189 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:24.190 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:24.190 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:24.190 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:24.190 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:24.190 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:24.191 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:24.192 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:24.192 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:24.192 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:24.192 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:24.193 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:24.193 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:24.930 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:25.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:26.935 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:27.937 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:28.190 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:28.192 7f75da7db0e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:28.193 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:28.193 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:28.193 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:28.193 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:28.194 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:28.194 7f75da7db0e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:28.195 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:28.195 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:28.195 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:28.195 7f75da7db0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:28.196 7f75da7db0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:28.196 7f75da7db0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:28.939 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:29.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:30.412 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:04:30.413 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:04:30.413 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:04:30.413 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:04:30.413 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:04:30.413 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:04:30.413 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:04:30.413 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:04:30.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:31.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:32.193 7f75da71b720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:32.259 7f75da71b720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:32.259 7f75da71b720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:32.259 7f75da71b720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:32.259 7f75da71b720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:32.259 7f75da71b720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:32.259 7f75da71b720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:32.260 7f75da71b720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:32.260 7f75da71b720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:32.260 7f75da71b720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:32.260 7f75da71b720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:32.260 7f75da71b720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:32.260 7f75da71b720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:32.260 7f75da71b720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:32.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:33.615 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:04:33.623 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:04:33.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:34.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:35.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:36.260 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:36.262 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:36.263 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:36.263 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:36.263 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:36.264 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:36.264 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:36.265 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:36.265 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:36.266 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:36.266 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:36.266 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:36.266 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:36.267 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:36.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:37.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:38.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:39.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:40.263 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:40.265 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:40.265 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:40.265 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:40.265 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:40.266 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:40.266 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:40.266 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:40.266 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:40.267 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:40.267 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:40.267 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:40.267 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:40.267 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:40.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:41.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:42.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:43.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:44.265 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:44.267 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:44.268 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:44.268 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:44.269 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:44.269 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:44.269 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:44.270 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:44.270 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:44.270 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:44.271 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:44.271 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:44.271 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:44.271 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:44.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:45.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:46.907 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:04:46.915 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:04:46.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:47.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:48.269 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:48.271 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:48.271 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:48.271 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:48.272 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:48.272 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:48.272 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:48.273 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:48.273 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:48.274 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:48.274 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:48.274 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:48.274 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:48.274 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:48.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:50.923 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:51.924 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:52.271 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:52.272 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:52.273 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:52.273 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:52.273 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:52.273 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:52.274 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:52.274 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:52.275 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:52.275 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:52.275 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:52.275 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:52.276 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:52.276 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:52.926 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:53.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:54.929 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:55.931 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:56.274 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:56.276 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:56.276 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:56.277 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:56.277 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:56.277 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:56.277 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:56.278 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:56.279 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:56.279 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:56.279 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:56.280 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:56.280 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:56.280 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:56.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:57.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:58.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:59.613 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:04:59.621 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:04:59.980 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:00.277 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:00.279 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:00.279 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:00.279 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:00.280 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:00.280 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:00.280 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:00.281 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:00.281 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:00.281 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:00.282 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:00.282 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:00.282 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:00.282 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:00.982 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:01.984 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:02.985 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:03.987 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:04.280 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:04.282 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:04.282 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:04.282 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:04.282 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:04.282 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:04.283 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:04.283 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:04.283 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:04.283 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:04.284 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:04.284 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:04.284 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:04.284 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:04.988 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:05.991 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:06.992 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:07.995 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:08.283 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:08.285 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:08.285 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:08.286 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:08.286 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:08.286 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:08.286 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:08.287 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:08.287 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:08.288 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:08.288 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:08.288 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:08.288 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:08.289 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:08.997 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:09.999 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:11.000 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:12.002 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:12.286 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:12.289 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:12.289 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:12.289 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:12.290 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:12.290 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:12.290 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:12.291 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:12.291 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:12.292 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:12.292 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:12.292 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:12.292 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:12.292 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:12.890 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:05:12.900 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:05:13.003 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:14.006 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:15.008 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:16.010 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:16.289 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:16.291 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:16.292 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:16.292 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:16.292 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:16.293 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:16.293 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:16.294 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:16.294 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:16.295 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:16.295 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:16.295 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:16.296 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:16.296 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:17.012 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:18.014 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:19.016 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:20.292 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:20.294 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:20.294 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:20.294 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:20.295 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:20.295 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:20.295 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:20.296 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:20.296 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:20.297 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:20.297 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:20.297 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:20.297 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:20.297 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:20.925 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:21.926 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:22.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:23.931 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:24.294 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:24.297 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:24.297 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:24.297 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:24.298 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:24.298 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:24.298 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:24.299 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:24.299 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:24.299 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:24.299 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:24.300 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:24.300 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:24.300 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:24.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:25.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:25.987 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:05:25.995 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:05:26.937 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:27.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:28.298 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:28.299 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:28.299 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:28.299 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:28.299 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:28.299 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:28.299 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:28.300 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:28.300 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:28.300 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:28.300 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:28.300 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:28.300 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:28.300 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:28.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:29.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:30.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:31.405 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:05:31.405 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:05:31.406 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:05:31.406 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:05:31.406 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:05:31.406 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:05:31.407 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:05:31.407 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:05:31.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:32.299 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:32.302 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:32.302 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:32.302 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:32.303 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:32.303 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:32.303 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:32.304 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:32.305 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:32.305 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:32.305 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:32.305 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:32.306 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:32.306 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:32.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:33.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:34.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:35.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:36.303 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:36.305 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:36.305 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:36.306 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:36.306 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:36.306 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:36.306 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:36.307 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:36.307 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:36.308 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:36.308 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:36.308 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:36.308 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:36.309 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:36.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:37.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:38.826 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:05:38.836 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:05:38.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:39.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:40.306 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:40.307 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:40.308 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:40.308 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:40.308 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:40.308 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:40.308 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:40.309 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:40.309 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:40.309 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:40.309 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:40.309 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:40.309 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:40.310 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:40.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:41.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:42.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:43.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:44.308 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:44.311 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:44.311 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:44.311 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:44.311 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:44.312 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:44.312 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:44.313 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:44.313 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:44.313 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:44.314 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:44.314 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:44.314 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:44.314 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:44.973 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:45.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:46.977 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:48.004 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:48.312 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:48.314 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:48.315 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:48.315 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:48.315 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:48.316 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:48.316 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:48.317 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:48.317 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:48.318 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:48.318 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:48.318 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:48.318 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:48.319 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:49.006 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:50.925 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:51.668 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:05:51.672 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:05:51.926 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:52.315 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:52.318 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:52.318 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:52.319 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:52.319 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:52.319 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:52.319 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:52.320 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:52.321 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:52.321 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:52.321 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:52.322 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:52.322 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:52.322 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:52.929 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:53.931 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:54.933 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:55.935 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:56.319 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:56.321 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:56.321 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:56.322 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:56.322 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:56.322 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:56.323 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:56.323 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:56.324 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:56.324 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:56.324 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:56.324 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:56.325 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:56.325 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:56.937 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:57.939 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:58.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:59.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:00.322 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:00.324 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:00.324 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:00.325 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:00.325 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:00.325 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:00.325 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:00.326 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:00.326 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:00.326 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:00.326 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:00.327 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:00.327 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:00.327 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:00.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:01.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:02.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:03.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:04.326 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:04.328 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:04.328 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:04.328 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:04.329 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:04.329 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:04.329 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:04.330 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:04.330 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:04.331 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:04.331 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:04.331 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:04.331 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:04.331 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:04.468 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:06:04.479 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:06:04.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:05.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:06.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:07.957 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:08.329 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:08.331 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:08.331 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:08.331 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:08.332 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:08.332 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:08.332 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:08.333 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:08.333 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:08.334 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:08.334 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:08.334 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:08.334 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:08.334 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:08.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:09.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:10.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:11.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:12.331 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:12.334 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:12.334 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:12.334 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:12.335 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:12.335 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:12.335 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:12.336 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:12.336 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:12.337 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:12.337 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:12.337 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:12.337 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:12.338 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:12.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:13.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:14.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:15.973 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:16.334 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:16.336 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:16.336 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:16.336 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:16.336 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:16.336 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:16.336 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:16.336 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:16.337 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:16.337 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:16.337 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:16.337 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:16.337 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:16.337 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:16.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:17.934 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:06:17.943 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:06:17.977 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:18.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:20.337 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:20.339 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:20.339 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:20.340 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:20.340 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:20.340 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:20.340 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:20.341 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:20.342 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:20.342 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:20.342 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:20.342 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:20.343 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:20.343 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:20.927 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:21.929 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:22.931 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:23.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:24.340 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:24.342 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:24.343 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:24.343 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:24.343 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:24.344 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:24.344 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:24.345 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:24.345 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:24.345 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:24.346 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:24.346 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:24.346 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:24.346 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:24.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:25.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:26.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:27.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:28.343 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:28.345 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:28.346 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:28.346 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:28.346 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:28.347 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:28.347 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:28.348 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:28.349 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:28.349 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:28.349 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:28.349 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:28.350 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:28.350 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:28.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:29.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:30.779 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:06:30.787 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:06:30.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:31.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:32.353 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:32.355 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:32.355 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:32.356 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:32.356 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:32.356 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:32.356 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:32.357 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:32.357 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:32.358 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:32.358 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:32.358 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:32.358 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:32.358 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:32.415 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:06:32.416 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:06:32.416 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:06:32.416 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:06:32.417 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:06:32.417 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:06:32.417 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:06:32.417 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:06:32.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:33.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:34.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:35.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:36.356 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:36.358 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:36.359 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:36.359 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:36.359 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:36.360 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:36.360 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:36.361 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:36.361 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:36.361 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:36.362 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:36.362 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:36.362 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:36.362 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:36.957 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:37.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:38.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:39.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:40.360 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:40.362 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:40.362 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:40.362 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:40.363 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:40.363 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:40.363 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:40.364 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:40.364 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:40.365 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:40.365 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:40.365 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:40.365 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:40.365 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:40.965 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:41.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:42.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:43.583 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:06:43.592 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:06:43.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:44.363 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:44.365 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:44.365 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:44.365 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:44.365 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:44.365 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:44.365 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:44.366 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:44.366 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:44.366 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:44.366 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:44.366 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:44.366 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:44.366 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:44.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:45.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:46.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:47.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:48.366 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:48.368 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:48.369 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:48.369 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:48.369 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:48.370 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:48.370 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:48.371 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:48.371 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:48.371 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:48.371 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:48.371 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:48.372 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:48.372 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:48.979 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:50.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:51.929 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:52.370 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:52.373 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:52.374 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:52.374 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:52.375 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:52.375 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:52.376 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:52.377 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:52.377 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:52.378 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:52.378 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:52.378 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:52.379 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:52.379 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:52.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:53.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:54.935 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:55.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:56.374 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:56.377 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:56.378 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:56.378 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:56.378 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:56.379 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:56.379 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:56.380 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:56.381 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:56.381 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:56.381 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:56.382 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:56.382 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:56.382 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:56.474 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:06:56.484 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:06:56.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:57.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:58.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:59.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:00.378 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:00.380 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:00.380 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:00.381 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:00.381 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:00.381 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:00.381 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:00.382 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:00.382 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:00.383 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:00.383 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:00.383 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:00.384 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:00.384 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:00.945 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:01.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:02.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:03.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:04.382 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:04.384 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:04.385 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:04.386 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:04.386 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:04.387 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:04.387 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:04.388 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:04.388 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:04.389 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:04.389 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:04.390 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:04.390 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:04.390 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:04.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:05.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:06.957 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:07.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:08.385 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:08.387 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:08.388 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:08.388 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:08.388 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:08.389 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:08.389 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:08.390 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:08.390 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:08.390 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:08.391 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:08.391 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:08.391 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:08.391 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:08.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:09.308 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:07:09.317 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:07:09.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:10.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:11.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:12.388 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:12.391 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:12.391 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:12.391 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:12.392 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:12.392 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:12.392 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:12.393 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:12.393 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:12.394 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:12.394 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:12.394 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:12.394 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:12.395 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:12.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:13.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:14.973 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:15.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:16.392 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:16.394 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:16.394 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:16.395 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:16.395 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:16.395 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:16.395 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:16.396 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:16.396 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:16.397 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:16.397 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:16.397 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:16.397 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:16.398 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:16.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:17.979 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:18.980 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:20.395 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:20.397 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:20.397 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:20.398 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:20.398 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:20.398 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:20.398 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:20.399 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:20.400 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:20.400 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:20.400 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:20.401 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:20.401 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:20.401 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:20.928 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:21.930 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:22.625 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:07:22.629 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:07:22.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:23.935 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:24.399 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:24.401 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:24.401 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:24.402 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:24.402 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:24.402 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:24.402 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:24.403 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:24.403 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:24.404 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:24.404 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:24.404 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:24.405 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:24.405 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:24.937 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:25.939 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:26.941 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:27.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:28.402 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:28.404 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:28.405 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:28.405 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:28.406 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:28.406 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:28.406 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:28.407 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:28.407 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:28.408 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:28.408 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:28.408 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:28.409 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:28.409 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:28.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:29.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:30.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:31.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:32.405 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:32.407 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:32.407 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:32.408 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:32.408 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:32.408 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:32.409 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:32.409 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:32.410 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:32.410 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:32.410 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:32.410 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:32.411 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:32.411 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:32.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:33.322 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:07:33.322 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:07:33.322 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:07:33.322 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:07:33.322 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:07:33.322 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:07:33.322 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:07:33.322 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:07:33.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:34.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:35.641 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:07:35.647 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:07:35.957 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:36.408 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:36.409 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:36.409 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:36.409 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:36.409 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:36.409 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:36.410 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:36.410 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:36.410 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:36.410 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:36.410 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:36.410 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:36.410 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:36.410 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:36.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:37.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:38.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:39.965 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:40.410 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:40.413 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:40.413 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:40.413 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:40.414 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:40.414 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:40.414 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:40.415 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:40.415 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:40.415 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:40.415 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:40.416 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:40.416 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:40.416 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:40.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:41.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:42.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:43.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:44.414 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:44.416 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:44.417 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:44.417 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:44.417 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:44.418 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:44.418 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:44.419 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:44.419 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:44.419 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:44.419 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:44.420 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:44.420 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:44.420 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:44.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:45.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:46.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:47.980 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:48.417 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:48.419 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:48.419 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:48.419 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:48.420 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:48.420 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:48.420 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:48.421 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:48.421 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:48.421 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:48.421 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:48.421 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:48.421 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:48.422 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:48.658 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:07:48.664 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:07:48.981 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:50.929 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:51.931 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:52.420 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:52.422 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:52.423 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:52.423 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:52.423 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:52.424 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:52.424 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:52.425 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:52.425 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:52.425 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:52.425 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:52.426 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:52.426 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:52.426 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:52.933 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:53.935 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:54.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:55.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:56.423 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:56.426 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:56.426 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:56.427 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:56.427 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:56.427 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:56.427 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:56.428 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:56.428 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:56.429 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:56.429 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:56.429 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:56.429 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:56.430 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:56.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:57.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:58.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:59.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:00.427 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:00.428 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:00.428 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:00.428 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:00.429 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:00.429 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:00.429 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:00.429 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:00.429 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:00.429 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:00.429 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:00.429 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:00.430 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:00.430 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:00.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:01.540 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:01.547 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:01.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:02.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:03.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:04.430 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:04.432 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:04.433 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:04.433 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:04.433 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:04.434 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:04.434 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:04.435 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:04.435 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:04.435 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:04.436 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:04.436 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:04.436 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:04.436 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:04.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:05.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:06.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:07.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:08.433 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:08.435 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:08.436 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:08.436 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:08.436 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:08.437 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:08.437 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:08.438 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:08.438 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:08.438 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:08.438 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:08.439 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:08.439 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:08.439 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:08.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:09.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:10.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:11.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:12.437 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:12.438 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:12.438 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:12.438 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:12.438 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:12.438 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:12.438 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:12.439 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:12.439 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:12.439 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:12.439 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:12.439 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:12.439 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:12.439 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:12.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:13.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:14.460 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:14.468 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:14.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:15.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:16.439 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:16.441 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:16.442 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:16.442 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:16.442 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:16.443 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:16.443 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:16.444 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:16.444 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:16.444 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:16.444 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:16.445 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:16.445 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:16.445 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:16.977 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:17.979 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:18.981 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:20.443 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:20.446 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:20.446 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:20.446 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:20.447 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:20.447 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:20.447 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:20.448 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:20.448 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:20.449 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:20.449 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:20.449 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:20.449 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:20.449 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:20.931 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:21.933 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:22.935 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:23.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:24.447 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:24.449 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:24.449 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:24.450 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:24.450 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:24.450 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:24.450 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:24.451 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:24.451 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:24.452 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:24.452 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:24.452 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:24.452 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:24.453 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:24.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:25.939 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:26.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:27.877 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:27.883 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:27.941 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:28.451 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:28.453 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:28.453 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:28.453 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:28.454 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:28.454 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:28.454 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:28.455 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:28.455 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:28.455 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:28.456 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:28.456 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:28.456 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:28.456 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:28.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:29.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:30.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:31.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:32.454 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:32.456 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:32.456 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:32.457 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:32.457 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:32.457 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:32.457 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:32.458 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:32.459 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:32.459 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:32.459 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:32.460 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:32.460 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:32.460 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:32.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:33.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:34.277 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:08:34.277 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:08:34.277 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:08:34.277 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:08:34.277 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:08:34.277 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:08:34.278 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:08:34.278 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:08:34.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:35.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:36.457 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:36.459 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:36.459 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:36.459 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:36.459 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:36.459 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:36.459 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:36.460 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:36.460 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:36.460 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:36.460 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:36.460 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:36.460 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:36.461 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:36.957 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:37.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:38.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:39.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:40.459 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:40.461 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:40.461 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:40.461 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:40.461 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:40.461 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:40.462 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:40.462 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:40.462 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:40.462 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:40.462 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:40.463 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:40.463 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:40.463 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:40.717 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:40.723 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:40.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:41.965 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:42.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:43.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:44.461 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:44.463 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:44.463 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:44.463 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:44.463 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:44.463 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:44.463 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:44.464 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:44.464 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:44.464 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:44.464 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:44.464 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:44.464 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:44.464 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:44.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:45.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:46.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:47.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:48.464 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:48.465 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:48.465 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:48.465 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:48.466 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:48.466 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:48.466 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:48.466 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:48.466 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:48.466 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:48.466 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:48.466 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:48.466 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:48.467 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:48.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:50.931 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:51.933 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:52.467 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:52.470 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:52.470 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:52.470 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:52.471 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:52.471 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:52.471 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:52.472 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:52.472 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:52.472 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:52.473 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:52.473 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:52.473 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:52.473 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:52.935 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:53.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:53.946 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:53.954 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:54.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:55.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:56.471 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:56.473 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:56.474 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:56.474 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:56.474 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:56.475 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:56.475 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:56.476 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:56.476 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:56.476 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:56.477 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:56.477 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:56.477 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:56.477 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:56.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:57.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:58.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:59.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:00.474 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:00.477 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:00.477 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:00.478 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:00.478 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:00.478 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:00.478 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:00.479 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:00.479 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:00.480 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:00.480 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:00.480 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:00.481 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:00.481 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:00.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:01.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:02.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:03.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:04.478 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:04.480 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:04.481 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:04.481 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:04.482 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:04.482 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:04.482 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:04.483 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:04.484 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:04.484 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:04.485 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:04.485 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:04.485 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:04.485 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:04.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:05.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:06.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:07.111 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:09:07.122 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:09:07.965 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:08.481 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:08.482 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:08.483 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:08.483 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:08.483 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:08.483 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:08.483 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:08.483 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:08.483 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:08.483 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:08.484 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:08.484 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:08.484 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:08.484 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:08.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:09.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:10.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:11.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:12.484 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:12.486 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:12.487 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:12.487 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:12.488 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:12.488 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:12.488 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:12.489 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:12.489 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:12.489 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:12.490 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:12.490 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:12.490 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:12.490 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:12.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:13.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:14.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:15.981 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:16.487 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:16.490 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:16.490 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:16.491 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:16.491 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:16.492 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:16.492 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:16.493 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:16.494 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:16.494 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:16.495 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:16.495 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:16.496 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:16.496 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:16.982 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:17.984 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:18.987 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:20.199 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:09:20.206 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:09:20.491 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:20.493 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:20.493 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:20.493 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:20.494 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:20.494 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:20.494 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:20.495 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:20.495 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:20.496 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:20.496 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:20.498 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:20.498 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:20.498 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:20.933 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:21.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:22.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:23.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:24.494 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:24.497 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:24.497 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:24.497 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:24.498 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:24.498 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:24.498 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:24.499 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:24.499 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:24.499 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:24.499 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:24.500 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:24.500 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:24.500 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:24.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:25.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:26.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:27.945 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:28.498 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:28.501 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:28.501 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:28.501 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:28.502 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:28.502 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:28.502 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:28.503 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:28.504 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:28.504 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:28.505 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:28.505 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:28.505 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:28.505 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:28.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:29.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:30.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:31.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:32.501 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:32.504 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:32.504 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:32.504 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:32.505 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:32.505 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:32.505 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:32.506 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:32.506 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:32.507 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:32.507 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:32.507 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:32.507 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:32.508 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:32.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:33.441 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:09:33.446 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:09:33.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:34.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:35.242 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:09:35.243 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:09:35.243 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:09:35.243 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:09:35.244 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:09:35.244 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:09:35.244 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:09:35.245 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:09:35.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:36.504 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:36.506 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:36.506 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:36.506 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:36.506 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:36.506 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:36.506 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:36.507 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:36.507 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:36.507 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:36.507 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:36.507 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:36.507 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:36.507 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:36.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:37.965 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:38.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:39.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:40.506 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:40.509 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:40.509 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:40.510 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:40.510 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:40.510 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:40.511 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:40.512 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:40.512 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:40.513 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:40.513 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:40.513 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:40.513 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:40.514 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:40.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:41.973 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:42.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:43.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:44.510 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:44.513 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:44.513 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:44.513 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:44.513 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:44.513 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:44.513 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:44.514 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:44.514 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:44.514 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:44.514 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:44.514 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:44.515 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:44.515 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:44.979 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:45.980 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:46.382 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:09:46.390 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:09:46.983 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:47.985 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:48.514 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:48.516 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:48.516 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:48.516 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:48.517 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:48.517 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:48.517 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:48.518 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:48.518 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:48.518 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:48.518 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:48.518 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:48.518 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:48.519 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:48.987 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:50.932 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:51.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:52.516 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:52.517 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:52.517 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:52.517 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:52.518 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:52.518 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:52.518 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:52.519 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:52.519 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:52.520 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:52.520 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:52.520 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:52.521 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:52.521 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:52.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:53.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:54.939 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:55.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:56.518 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:56.520 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:56.521 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:56.521 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:56.521 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:56.521 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:56.522 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:56.522 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:56.523 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:56.523 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:56.523 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:56.523 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:56.524 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:56.524 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:56.941 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:57.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:58.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:59.384 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:09:59.391 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:09:59.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:00.522 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:00.524 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:00.524 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:00.525 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:00.525 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:00.525 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:00.526 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:00.526 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:00.527 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:00.527 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:00.527 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:00.527 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:00.528 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:00.528 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:00.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:01.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:02.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:03.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:04.525 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:04.527 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:04.528 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:04.528 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:04.529 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:04.529 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:04.529 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:04.530 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:04.530 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:04.531 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:04.531 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:04.531 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:04.531 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:04.532 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:04.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:05.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:06.957 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:07.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:08.529 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:08.531 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:08.532 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:08.532 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:08.532 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:08.533 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:08.533 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:08.534 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:08.534 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:08.534 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:08.535 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:08.535 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:08.535 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:08.535 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:08.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:09.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:10.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:11.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:12.349 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:10:12.357 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:10:12.532 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:12.534 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:12.534 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:12.535 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:12.535 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:12.535 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:12.535 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:12.536 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:12.536 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:12.537 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:12.537 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:12.537 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:12.538 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:12.538 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:12.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:13.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:14.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:15.973 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:16.535 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:16.537 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:16.537 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:16.537 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:16.538 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:16.538 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:16.538 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:16.539 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:16.539 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:16.539 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:16.539 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:16.539 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:16.539 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:16.540 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:16.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:17.977 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:18.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:20.538 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:20.540 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:20.541 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:20.541 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:20.541 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:20.542 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:20.542 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:20.543 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:20.543 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:20.543 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:20.543 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:20.544 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:20.544 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:20.544 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:20.934 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:21.935 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:22.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:23.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:24.542 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:24.543 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:24.543 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:24.543 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:24.544 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:24.544 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:24.544 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:24.544 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:24.544 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:24.545 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:24.545 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:24.545 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:24.545 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:24.545 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:24.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:25.223 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:10:25.232 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:10:25.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:26.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:27.945 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:28.545 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:28.547 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:28.547 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:28.547 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:28.548 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:28.548 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:28.548 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:28.549 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:28.549 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:28.550 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:28.550 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:28.550 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:28.550 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:28.551 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:28.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:29.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:30.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:31.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:32.548 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:32.549 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:32.549 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:32.549 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:32.550 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:32.550 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:32.550 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:32.550 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:32.550 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:32.551 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:32.551 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:32.551 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:32.551 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:32.551 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:32.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:33.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:34.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:35.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:36.246 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:10:36.247 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:10:36.247 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:10:36.247 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:10:36.247 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:10:36.248 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:10:36.248 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:10:36.248 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:10:36.550 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:36.553 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:36.553 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:36.553 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:36.554 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:36.554 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:36.554 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:36.555 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:36.555 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:36.555 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:36.556 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:36.556 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:36.556 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:36.556 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:36.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:37.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:38.568 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:10:38.577 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:10:38.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:39.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:40.554 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:40.556 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:40.557 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:40.557 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:40.557 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:40.557 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:40.558 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:40.558 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:40.559 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:40.559 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:40.559 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:40.559 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:40.560 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:40.560 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:40.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:41.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:42.973 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:43.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:44.558 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:44.560 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:44.561 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:44.561 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:44.561 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:44.561 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:44.562 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:44.563 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:44.563 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:44.563 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:44.563 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:44.564 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:44.564 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:44.564 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:44.977 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:45.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:46.981 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:47.982 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:48.562 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:48.564 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:48.564 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:48.564 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:48.565 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:48.565 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:48.565 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:48.566 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:48.566 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:48.567 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:48.567 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:48.567 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:48.567 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:48.568 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:48.984 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:50.936 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:51.452 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:10:51.459 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:10:51.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:52.565 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:52.566 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:52.567 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:52.567 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:52.567 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:52.567 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:52.568 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:52.568 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:52.569 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:52.569 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:52.569 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:52.569 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:52.569 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:52.570 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:52.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:53.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:54.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:55.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:56.568 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:56.570 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:56.570 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:56.571 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:56.571 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:56.571 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:56.571 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:56.572 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:56.572 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:56.573 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:56.573 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:56.573 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:56.573 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:56.573 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:56.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:57.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:58.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:59.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:00.571 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:00.572 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:00.572 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:00.572 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:00.572 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:00.572 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:00.572 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:00.573 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:00.573 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:00.573 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:00.573 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:00.573 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:00.573 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:00.573 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:00.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:01.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:02.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:03.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:04.176 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:04.183 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:04.574 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:04.576 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:04.576 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:04.577 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:04.577 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:04.577 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:04.577 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:04.578 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:04.579 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:04.579 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:04.579 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:04.579 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:04.580 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:04.580 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:04.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:05.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:06.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:07.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:08.577 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:08.579 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:08.580 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:08.580 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:08.581 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:08.581 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:08.581 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:08.582 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:08.582 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:08.582 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:08.582 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:08.583 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:08.583 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:08.583 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:08.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:09.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:10.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:11.977 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:12.580 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:12.583 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:12.584 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:12.584 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:12.585 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:12.585 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:12.585 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:12.586 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:12.587 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:12.587 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:12.587 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:12.588 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:12.588 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:12.588 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:12.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:13.981 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:14.983 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:15.985 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:16.584 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:16.585 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:16.586 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:16.586 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:16.586 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:16.586 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:16.586 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:16.586 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:16.587 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:16.587 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:16.587 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:16.587 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:16.587 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:16.587 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:16.987 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:17.092 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:17.098 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:17.989 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:18.991 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:20.587 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:20.590 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:20.590 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:20.590 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:20.591 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:20.591 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:20.591 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:20.592 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:20.592 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:20.593 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:20.593 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:20.593 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:20.593 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:20.594 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:20.937 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:21.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:22.941 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:23.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:24.591 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:24.594 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:24.594 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:24.594 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:24.595 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:24.595 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:24.595 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:24.596 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:24.596 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:24.597 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:24.597 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:24.597 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:24.597 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:24.598 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:24.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:25.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:26.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:27.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:28.594 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:28.596 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:28.597 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:28.597 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:28.597 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:28.598 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:28.598 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:28.599 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:28.599 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:28.599 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:28.600 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:28.600 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:28.600 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:28.601 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:28.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:29.931 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:29.936 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:29.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:30.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:31.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:32.598 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:32.599 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:32.599 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:32.600 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:32.600 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:32.600 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:32.600 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:32.600 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:32.600 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:32.600 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:32.600 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:32.601 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:32.601 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:32.601 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:32.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:33.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:34.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:35.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:36.600 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:36.603 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:36.604 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:36.604 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:36.605 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:36.605 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:36.605 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:36.606 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:36.607 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:36.607 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:36.608 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:36.608 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:36.609 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:36.609 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:36.965 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:37.380 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:11:37.381 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:11:37.381 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:11:37.382 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:11:37.382 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:11:37.382 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:11:37.382 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:11:37.383 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:11:37.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:38.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:39.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:40.604 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:40.607 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:40.607 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:40.607 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:40.608 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:40.608 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:40.608 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:40.609 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:40.609 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:40.610 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:40.610 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:40.610 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:40.610 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:40.611 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:40.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:41.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:42.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:43.171 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:43.179 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:43.977 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:44.608 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:44.610 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:44.610 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:44.611 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:44.611 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:44.611 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:44.611 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:44.612 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:44.612 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:44.613 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:44.613 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:44.613 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:44.613 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:44.614 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:44.979 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:45.980 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:46.982 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:47.984 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:48.612 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:48.614 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:48.614 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:48.615 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:48.615 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:48.616 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:48.616 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:48.617 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:48.617 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:48.618 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:48.618 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:48.618 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:48.618 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:48.619 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:48.985 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:50.938 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:51.939 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:52.615 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:52.618 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:52.618 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:52.618 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:52.619 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:52.619 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:52.620 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:52.620 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:52.621 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:52.621 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:52.621 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:52.621 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:52.622 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:52.622 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:52.941 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:53.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:54.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:55.945 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:56.201 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:56.208 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:56.619 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:56.621 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:56.622 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:56.622 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:56.622 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:56.623 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:56.623 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:56.624 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:56.624 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:56.624 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:56.624 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:56.625 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:56.625 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:56.625 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:56.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:57.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:58.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:59.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:00.623 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:00.625 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:00.626 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:00.626 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:00.626 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:00.627 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:00.627 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:00.628 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:00.628 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:00.629 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:00.629 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:00.629 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:00.629 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:00.630 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:00.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:01.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:02.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:03.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:04.626 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:04.628 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:04.629 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:04.629 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:04.629 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:04.629 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:04.630 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:04.630 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:04.631 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:04.631 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:04.631 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:04.631 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:04.632 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:04.632 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:04.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:05.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:06.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:07.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:08.629 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:08.631 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:08.631 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:08.631 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:08.632 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:08.632 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:08.632 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:08.633 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:08.633 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:08.634 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:08.634 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:08.634 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:08.634 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:08.635 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:08.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:09.448 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:12:09.456 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:12:09.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:10.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:11.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:12.632 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:12.634 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:12.635 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:12.635 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:12.635 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:12.636 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:12.636 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:12.637 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:12.637 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:12.637 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:12.637 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:12.638 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:12.638 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:12.638 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:12.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:13.980 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:14.982 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:15.983 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:16.636 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:16.637 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:16.637 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:16.637 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:16.637 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:16.637 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:16.638 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:16.638 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:16.638 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:16.638 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:16.638 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:16.638 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:16.638 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:16.638 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:16.985 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:17.988 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:18.989 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:20.638 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:20.640 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:20.641 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:20.641 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:20.642 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:20.642 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:20.642 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:20.643 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:20.643 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:20.644 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:20.644 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:20.644 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:20.644 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:20.645 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:20.939 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:21.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:22.505 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:12:22.514 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:12:22.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:23.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:24.641 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:24.643 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:24.643 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:24.644 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:24.644 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:24.644 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:24.644 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:24.645 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:24.645 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:24.646 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:24.646 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:24.646 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:24.646 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:24.646 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:24.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:25.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:26.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:27.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:28.644 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:28.647 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:28.647 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:28.647 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:28.648 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:28.648 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:28.648 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:28.649 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:28.649 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:28.650 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:28.650 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:28.650 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:28.651 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:28.651 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:28.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:29.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:30.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:31.957 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:32.648 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:32.650 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:32.651 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:32.651 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:32.651 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:32.651 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:32.651 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:32.652 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:32.652 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:32.652 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:32.652 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:32.652 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:32.652 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:32.652 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:32.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:33.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:34.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:35.502 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:12:35.512 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:12:35.965 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:36.651 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:36.653 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:36.654 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:36.654 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:36.654 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:36.654 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:36.655 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:36.655 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:36.656 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:36.656 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:36.656 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:36.657 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:36.657 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:36.657 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:36.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:37.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:38.349 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:12:38.349 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:12:38.350 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:12:38.350 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:12:38.350 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:12:38.350 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:12:38.351 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:12:38.351 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:12:38.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:39.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:40.655 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:40.657 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:40.658 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:40.658 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:40.658 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:40.658 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:40.659 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:40.659 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:40.660 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:40.660 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:40.660 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:40.660 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:40.661 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:40.661 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:40.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:41.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:42.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:43.980 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:44.659 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:44.661 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:44.661 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:44.661 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:44.661 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:44.662 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:44.662 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:44.663 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:44.663 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:44.663 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:44.663 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:44.663 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:44.664 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:44.664 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:44.982 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:45.983 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:46.985 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:47.987 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:48.661 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:48.663 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:48.664 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:48.664 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:48.664 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:48.664 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:48.665 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:48.665 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:48.666 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:48.666 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:48.666 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:48.666 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:48.667 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:48.667 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:48.930 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:12:48.936 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:12:48.989 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:50.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:51.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:52.665 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:52.667 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:52.668 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:52.668 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:52.669 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:52.669 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:52.670 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:52.671 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:52.671 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:52.672 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:52.672 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:52.672 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:52.673 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:52.673 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:52.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:53.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:54.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:55.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:56.668 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:56.670 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:56.671 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:56.671 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:56.671 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:56.671 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:56.672 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:56.672 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:56.673 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:56.673 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:56.673 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:56.673 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:56.674 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:56.674 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:56.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:57.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:58.957 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:59.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:00.671 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:00.673 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:00.674 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:00.674 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:00.675 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:00.675 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:00.675 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:00.676 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:00.676 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:00.676 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:00.677 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:00.677 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:00.677 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:00.677 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:00.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:01.800 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:01.805 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:13:01.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:02.965 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:03.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:04.675 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:04.677 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:04.677 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:04.677 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:04.678 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:04.678 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:04.678 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:04.679 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:04.679 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:04.680 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:04.680 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:04.680 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:04.680 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:04.680 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:04.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:05.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:06.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:07.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:08.679 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:08.681 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:08.681 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:08.681 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:08.682 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:08.682 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:08.682 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:08.683 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:08.683 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:08.684 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:08.684 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:08.684 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:08.684 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:08.685 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:08.977 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:09.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:10.981 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:11.983 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:12.683 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:12.685 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:12.686 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:12.686 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:12.687 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:12.687 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:12.687 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:12.688 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:12.688 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:12.688 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:12.689 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:12.689 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:12.689 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:12.689 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:12.985 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:13.987 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:14.894 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:14.902 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:13:14.988 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:15.991 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:16.687 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:16.690 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:16.690 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:16.690 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:16.691 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:16.691 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:16.692 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:16.692 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:16.693 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:16.693 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:16.694 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:16.694 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:16.694 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:16.694 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:16.993 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:17.995 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:18.996 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:20.691 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:20.694 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:20.694 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:20.694 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:20.695 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:20.695 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:20.695 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:20.696 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:20.696 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:20.697 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:20.697 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:20.697 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:20.697 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:20.697 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:20.940 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:21.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:22.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:23.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:24.695 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:24.698 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:24.698 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:24.698 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:24.699 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:24.699 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:24.700 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:24.701 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:24.701 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:24.701 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:24.701 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:24.702 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:24.702 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:24.702 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:24.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:25.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:26.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:27.880 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:27.889 7f75daa70d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:13:27.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:28.699 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:28.702 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:28.702 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:28.702 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:28.703 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:28.703 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:28.703 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:28.704 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:28.704 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:28.705 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:28.705 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:28.705 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:28.705 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:28.706 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:28.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:29.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:30.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:31.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:32.703 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:32.705 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:32.706 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:32.706 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:32.706 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:32.707 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:32.707 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:32.708 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:32.708 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:32.708 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:32.708 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:32.709 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:32.709 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:32.709 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:32.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:33.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:34.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:35.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:36.707 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:36.710 7f75daa70d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:36.710 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:36.710 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:36.711 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:36.711 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:36.711 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:36.712 7f75daa70d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:36.713 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:36.713 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:36.713 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:36.713 7f75daa70d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:36.714 7f75daa70d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:36.714 7f75daa70d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:36.973 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:37.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:38.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:39.365 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:13:39.366 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:13:39.366 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:13:39.366 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:13:39.366 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:13:39.366 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:13:39.366 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:13:39.366 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:13:39.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:40.711 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:40.713 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:40.713 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:40.714 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:40.714 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:40.714 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:40.714 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:40.715 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:40.716 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:40.716 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:40.716 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:40.717 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:40.717 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:40.717 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:40.979 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:41.015 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:41.025 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:13:41.982 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:42.984 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:43.986 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:44.714 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:44.716 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:44.716 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:44.716 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:44.717 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:44.717 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:44.717 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:44.717 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:44.717 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:44.718 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:44.718 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:44.718 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:44.718 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:44.718 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:44.987 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:45.989 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:46.991 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:47.994 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:48.717 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:48.720 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:48.720 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:48.720 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:48.721 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:48.721 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:48.721 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:48.722 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:48.722 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:48.723 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:48.723 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:48.723 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:48.723 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:48.724 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:48.996 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:50.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:51.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:52.722 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:52.724 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:52.724 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:52.724 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:52.725 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:52.725 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:52.725 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:52.727 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:52.727 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:52.727 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:52.727 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:52.728 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:52.728 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:52.728 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:52.945 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:53.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:54.341 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:54.351 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:13:54.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:55.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:56.724 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:56.727 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:56.727 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:56.727 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:56.728 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:56.728 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:56.728 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:56.729 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:56.729 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:56.729 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:56.730 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:56.730 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:56.730 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:56.730 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:56.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:57.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:58.955 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:59.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:00.727 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:00.729 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:00.730 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:00.730 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:00.730 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:00.731 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:00.731 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:00.732 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:00.732 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:00.732 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:00.732 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:00.733 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:00.733 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:00.733 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:00.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:01.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:02.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:03.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:04.730 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:04.732 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:04.733 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:04.733 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:04.734 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:04.734 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:04.734 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:04.735 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:04.735 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:04.735 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:04.735 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:04.736 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:04.736 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:04.736 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:04.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:05.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:06.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:07.175 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:07.184 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:07.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:08.734 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:08.737 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:08.737 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:08.738 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:08.738 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:08.738 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:08.738 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:08.739 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:08.739 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:08.739 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:08.739 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:08.739 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:08.739 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:08.740 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:08.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:09.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:10.977 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:11.979 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:12.738 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:12.740 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:12.741 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:12.741 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:12.741 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:12.742 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:12.742 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:12.743 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:12.743 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:12.744 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:12.744 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:12.744 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:12.744 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:12.745 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:12.981 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:13.984 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:14.985 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:15.986 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:16.742 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:16.744 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:16.745 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:16.745 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:16.745 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:16.745 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:16.745 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:16.745 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:16.746 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:16.746 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:16.746 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:16.746 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:16.746 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:16.746 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:16.988 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:17.990 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:18.992 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:20.278 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:20.286 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:20.745 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:20.747 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:20.747 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:20.747 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:20.747 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:20.747 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:20.747 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:20.748 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:20.748 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:20.748 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:20.748 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:20.748 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:20.748 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:20.748 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:20.942 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:21.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:22.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:23.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:24.748 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:24.750 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:24.751 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:24.751 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:24.751 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:24.752 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:24.752 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:24.753 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:24.753 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:24.753 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:24.753 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:24.754 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:24.754 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:24.754 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:24.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:25.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:26.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:27.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:28.751 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:28.753 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:28.753 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:28.753 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:28.753 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:28.753 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:28.753 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:28.753 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:28.754 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:28.754 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:28.754 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:28.754 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:28.754 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:28.754 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:28.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:29.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:30.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:31.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:32.753 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:32.755 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:32.755 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:32.756 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:32.756 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:32.756 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:32.756 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:32.757 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:32.757 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:32.758 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:32.758 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:32.758 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:32.758 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:32.759 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:32.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:33.326 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:33.340 7f75daa704a0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:33.965 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:34.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:35.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:36.757 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:36.759 7f75daa704a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:36.759 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:36.759 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:36.760 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:36.760 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:36.760 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:36.761 7f75daa704a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:36.761 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:36.762 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:36.762 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:36.762 7f75daa704a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:36.762 7f75daa704a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:36.763 7f75daa704a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:36.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:37.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:38.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:39.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:40.328 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:14:40.329 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:14:40.329 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:14:40.329 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:14:40.330 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:14:40.330 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:14:40.330 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:14:40.330 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:14:40.760 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:40.762 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:40.762 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:40.762 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:40.763 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:40.763 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:40.763 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:40.764 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:40.764 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:40.764 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:40.765 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:40.765 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:40.765 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:40.765 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:40.977 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:41.979 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:42.981 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:43.983 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:44.763 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:44.766 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:44.766 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:44.766 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:44.767 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:44.767 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:44.767 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:44.768 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:44.768 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:44.768 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:44.769 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:44.769 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:44.769 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:44.769 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:44.984 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:45.985 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:46.175 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:46.182 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:46.987 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:47.988 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:48.767 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:48.769 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:48.770 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:48.770 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:48.770 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:48.770 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:48.771 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:48.771 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:48.772 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:48.772 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:48.772 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:48.772 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:48.773 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:48.773 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:48.991 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:50.943 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:51.945 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:52.771 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:52.773 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:52.773 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:52.774 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:52.774 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:52.774 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:52.774 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:52.775 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:52.776 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:52.776 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:52.777 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:52.777 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:52.777 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:52.778 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:52.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:53.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:54.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:55.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:56.775 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:56.778 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:56.778 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:56.778 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:56.779 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:56.779 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:56.780 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:56.781 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:56.781 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:56.782 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:56.782 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:56.783 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:56.783 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:56.783 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:56.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:57.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:58.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:59.806 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:59.811 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:59.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:00.779 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:00.781 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:00.781 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:00.781 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:00.782 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:00.782 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:00.782 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:00.782 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:00.783 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:00.783 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:00.783 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:00.783 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:00.783 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:00.783 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:00.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:01.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:02.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:03.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:04.782 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:04.783 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:04.784 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:04.784 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:04.784 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:04.784 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:04.784 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:04.785 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:04.785 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:04.785 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:04.785 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:04.785 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:04.785 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:04.785 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:04.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:05.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:06.973 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:07.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:08.784 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:08.785 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:08.786 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:08.786 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:08.786 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:08.786 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:08.786 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:08.787 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:08.787 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:08.787 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:08.787 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:08.787 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:08.787 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:08.788 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:08.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:09.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:10.980 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:12.044 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:12.787 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:12.791 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:12.791 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:12.791 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:12.792 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:12.792 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:12.792 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:12.793 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:12.793 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:12.794 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:12.794 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:12.794 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:12.794 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:12.794 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:12.902 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:15:12.912 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:15:13.046 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:14.047 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:15.048 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:16.050 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:16.792 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:16.795 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:16.795 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:16.795 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:16.796 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:16.796 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:16.796 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:16.797 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:16.797 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:16.798 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:16.798 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:16.798 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:16.798 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:16.799 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:17.053 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:18.055 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:19.056 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:20.796 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:20.799 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:20.799 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:20.799 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:20.800 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:20.800 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:20.801 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:20.802 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:20.802 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:20.803 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:20.803 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:20.803 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:20.804 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:20.804 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:20.944 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:21.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:22.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:23.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:24.800 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:24.802 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:24.802 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:24.803 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:24.803 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:24.803 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:24.803 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:24.804 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:24.804 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:24.805 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:24.805 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:24.805 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:24.805 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:24.806 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:24.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:25.783 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:15:25.792 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:15:25.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:26.957 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:27.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:28.803 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:28.806 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:28.806 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:28.806 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:28.807 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:28.807 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:28.807 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:28.808 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:28.809 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:28.809 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:28.810 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:28.810 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:28.810 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:28.810 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:28.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:29.961 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:30.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:31.963 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:32.806 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:32.808 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:32.808 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:32.808 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:32.809 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:32.809 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:32.809 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:32.809 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:32.810 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:32.810 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:32.810 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:32.810 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:32.810 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:32.810 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:32.965 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:33.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:34.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:35.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:36.810 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:36.812 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:36.812 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:36.812 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:36.813 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:36.813 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:36.813 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:36.814 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:36.814 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:36.815 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:36.815 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:36.815 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:36.815 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:36.815 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:36.973 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:37.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:38.697 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:15:38.707 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:15:38.977 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:39.979 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:40.812 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:40.816 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:40.816 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:40.817 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:40.817 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:40.817 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:40.817 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:40.818 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:40.818 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:40.819 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:40.819 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:40.819 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:40.819 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:40.819 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:40.980 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:41.260 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:15:41.260 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:15:41.261 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:15:41.261 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:15:41.261 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:15:41.261 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:15:41.261 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:15:41.261 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:15:41.982 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:42.984 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:43.986 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:44.817 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:44.819 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:44.820 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:44.820 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:44.821 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:44.821 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:44.821 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:44.822 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:44.822 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:44.823 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:44.823 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:44.823 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:44.823 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:44.823 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:44.988 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:45.990 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:46.992 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:47.994 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:48.821 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:48.823 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:48.824 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:48.824 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:48.824 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:48.825 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:48.825 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:48.826 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:48.826 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:48.826 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:48.826 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:48.827 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:48.827 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:48.827 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:48.997 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:50.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:51.751 7f75daa70040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:15:51.759 7f75daa70040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:15:51.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:52.824 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:52.825 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:52.825 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:52.825 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:52.826 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:52.826 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:52.826 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:52.826 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:52.826 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:52.826 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:52.826 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:52.826 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:52.826 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:52.826 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:52.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:53.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:54.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:55.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:56.827 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:56.829 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:56.829 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:56.829 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:56.830 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:56.830 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:56.830 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:56.831 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:56.831 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:56.832 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:56.832 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:56.832 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:56.832 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:56.832 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:56.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:57.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:58.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:59.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:00.830 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:00.831 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:00.832 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:00.832 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:00.832 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:00.832 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:00.832 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:00.832 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:00.833 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:00.833 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:00.833 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:00.833 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:00.833 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:00.833 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:00.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:01.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:02.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:03.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:04.832 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:04.835 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:04.835 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:04.835 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:04.836 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:04.836 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:04.836 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:04.837 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:04.837 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:04.837 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:04.838 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:04.838 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:04.838 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:04.838 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:04.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:05.185 7f75daa70040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:16:05.193 7f75daa70040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:16:05.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:06.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:07.979 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:08.837 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:08.839 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:08.839 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:08.840 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:08.840 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:08.840 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:08.840 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:08.841 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:08.841 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:08.842 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:08.842 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:08.842 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:08.843 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:08.843 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:08.981 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:09.983 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:10.985 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:11.987 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:12.840 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:12.843 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:12.843 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:12.843 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:12.844 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:12.844 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:12.844 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:12.845 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:12.845 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:12.846 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:12.846 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:12.846 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:12.846 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:12.847 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:12.989 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:13.991 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:14.993 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:15.995 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:16.844 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:16.845 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:16.845 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:16.845 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:16.845 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:16.846 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:16.846 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:16.846 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:16.846 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:16.846 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:16.846 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:16.846 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:16.846 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:16.847 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:16.997 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:17.999 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:18.069 7f75daa70040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:16:18.073 7f75daa70040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:16:19.000 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:20.847 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:20.849 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:20.849 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:20.849 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:20.850 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:20.850 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:20.850 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:20.850 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:20.851 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:20.851 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:20.851 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:20.851 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:20.851 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:20.852 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:20.946 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:21.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:22.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:23.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:24.851 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:24.853 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:24.853 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:24.854 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:24.854 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:24.854 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:24.854 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:24.855 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:24.855 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:24.856 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:24.856 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:24.856 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:24.856 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:24.857 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:24.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:25.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:26.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:27.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:28.854 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:28.856 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:28.857 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:28.857 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:28.858 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:28.858 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:28.858 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:28.859 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:28.859 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:28.860 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:28.860 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:28.860 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:28.860 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:28.861 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:28.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:29.965 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:30.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:31.012 7f75daa70040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:16:31.019 7f75daa70040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:16:31.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:32.858 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:32.860 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:32.861 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:32.861 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:32.862 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:32.862 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:32.862 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:32.863 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:32.863 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:32.864 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:32.864 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:32.864 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:32.865 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:32.865 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:32.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:33.972 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:34.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:35.977 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:36.862 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:36.864 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:36.864 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:36.865 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:36.865 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:36.865 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:36.866 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:36.866 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:36.867 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:36.867 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:36.867 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:36.868 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:36.868 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:36.868 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:36.979 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:37.981 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:38.983 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:39.985 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:40.865 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:40.867 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:40.867 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:40.867 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:40.867 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:40.868 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:40.868 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:40.869 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:40.869 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:40.869 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:40.869 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:40.869 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:40.870 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:40.870 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:40.987 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:41.988 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:42.161 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:16:42.161 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:16:42.161 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:16:42.161 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:16:42.161 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:16:42.161 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:16:42.161 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:16:42.161 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:16:42.989 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:43.991 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:44.041 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:16:44.047 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:16:44.868 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:44.870 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:44.870 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:44.870 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:44.870 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:44.870 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:44.870 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:44.871 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:44.871 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:44.871 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:44.871 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:44.871 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:44.871 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:44.872 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:44.993 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:45.995 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:46.998 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:48.000 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:48.871 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:48.873 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:48.873 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:48.873 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:48.874 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:48.874 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:48.874 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:48.874 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:48.875 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:48.875 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:48.875 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:48.875 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:48.875 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:48.875 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:49.002 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:50.947 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:51.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:52.875 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:52.877 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:52.878 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:52.878 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:52.879 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:52.879 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:52.879 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:52.880 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:52.880 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:52.881 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:52.881 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:52.881 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:52.881 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:52.881 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:52.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:53.952 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:54.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:55.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:56.879 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:56.881 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:56.882 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:56.882 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:56.882 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:56.882 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:56.883 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:56.883 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:56.884 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:56.884 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:56.884 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:56.884 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:56.885 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:16:56.885 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:56.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:57.019 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:16:57.028 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:16:57.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:58.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:16:59.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:00.882 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:00.885 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:00.885 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:00.886 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:00.886 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:00.887 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:00.887 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:00.888 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:00.888 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:00.889 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:00.889 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:00.889 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:00.890 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:00.890 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:00.965 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:01.967 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:02.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:03.970 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:04.886 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:04.888 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:04.889 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:04.889 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:04.890 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:04.890 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:04.890 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:04.891 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:04.891 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:04.891 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:04.891 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:04.892 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:04.892 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:04.892 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:04.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:05.974 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:06.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:07.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:08.890 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:08.892 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:08.892 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:08.892 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:08.892 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:08.893 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:08.893 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:08.894 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:08.894 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:08.894 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:08.894 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:08.895 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:08.895 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:08.895 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:08.980 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:09.981 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:10.467 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:17:10.476 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:17:10.983 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:11.984 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:12.892 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:12.894 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:12.895 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:12.895 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:12.895 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:12.896 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:12.896 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:12.897 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:12.897 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:12.898 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:12.898 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:12.898 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:12.898 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:12.898 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:12.986 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:13.449 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:17:13.458 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:17:13.988 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:14.990 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:15.992 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:16.374 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:17:16.383 7f75daa70400 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:17:16.895 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:16.898 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:16.898 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:16.898 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:16.898 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:16.899 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:16.899 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:16.900 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:16.900 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:16.900 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:16.901 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:16.901 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:16.901 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:16.901 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:16.993 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:17.996 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:18.998 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:20.899 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:20.901 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:20.902 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:20.902 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:20.902 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:20.903 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:20.903 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:20.903 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:20.904 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:20.904 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:20.904 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:20.905 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:20.905 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:20.905 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:20.948 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:21.949 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:22.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:23.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:24.902 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:24.904 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:24.905 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:24.905 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:24.905 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:24.906 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:24.906 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:24.907 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:24.907 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:24.908 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:24.908 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:24.908 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:24.908 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:24.908 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:24.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:25.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:26.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:27.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:28.906 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:28.907 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:28.907 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:28.907 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:28.907 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:28.907 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:28.908 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:28.908 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:28.908 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:28.908 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:28.908 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:28.908 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:28.908 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:28.908 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:28.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:29.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:30.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:31.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:32.908 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:32.910 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:32.910 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:32.911 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:32.911 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:32.911 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:32.911 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:32.912 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:32.913 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:32.913 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:32.913 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:32.913 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:32.914 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:32.914 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:32.968 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:33.969 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:34.971 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:35.973 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:36.911 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:36.912 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:36.913 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:36.913 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:36.913 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:36.913 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:36.913 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:36.913 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:36.914 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:36.914 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:36.914 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:36.914 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:36.914 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:36.914 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:36.975 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:37.976 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:38.978 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:39.980 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:40.914 7f75daa70400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:40.917 7f75daa70400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:40.917 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:40.918 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:40.918 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:40.918 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:40.918 7f75daa70400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:40.919 7f75daa70400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:40.919 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:40.920 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:40.920 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:40.920 7f75daa70400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:40.921 7f75daa70400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:40.921 7f75daa70400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:40.982 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:41.984 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:42.986 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:43.147 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:17:43.148 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:17:43.148 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:17:43.148 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:17:43.148 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:17:43.148 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:17:43.148 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:17:43.148 7f75da7db900 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:17:43.988 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:44.918 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:44.921 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:44.921 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:44.922 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:44.922 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:44.922 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:44.922 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:44.923 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:44.924 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:44.924 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:44.925 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:44.925 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:44.925 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:44.925 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:44.989 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:45.991 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:46.993 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:47.537 7f75daa70040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:17:47.545 7f75daa70040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:17:47.995 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:48.921 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:48.923 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:48.924 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:48.924 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:48.924 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:48.924 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:48.925 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:48.925 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:48.926 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:48.926 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:48.926 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:48.926 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:48.927 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:48.927 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:48.997 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:50.950 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:51.951 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:52.924 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:52.926 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:52.927 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:52.927 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:52.928 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:52.928 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:52.928 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:52.929 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:52.929 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:52.930 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:52.930 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:52.930 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:52.930 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:52.930 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:52.953 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:53.954 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:54.956 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:55.958 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:56.928 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:56.930 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:56.931 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:56.931 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:56.931 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:56.932 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:56.932 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:56.933 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:56.933 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:56.933 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:56.933 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:56.934 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:56.934 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:17:56.934 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:56.959 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:57.960 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:58.962 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:17:59.964 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:18:00.931 7f75daa70040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:00.933 7f75daa70040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:00.933 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:00.933 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:00.934 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:00.934 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:00.934 7f75daa70040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:00.935 7f75daa70040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:00.935 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:00.936 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.56:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:00.936 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:00.936 7f75daa70040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:00.936 7f75daa70040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691835793571 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:18:00.936 7f75daa70040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:00.966 7f75daa57720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process.