04/20/2026 05:25:24.739 7f74065d3898 [contrail-topology] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 04/20/2026 05:25:24.739 7f74065d3898 [contrail-topology] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 04/20/2026 05:25:24.739 7f74065d3898 [contrail-topology] [INFO]: SANDESH: Flow Logging: [None] -> [False] 04/20/2026 05:25:24.739 7f74065d3898 [contrail-topology] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map 04/20/2026 05:25:24.739 7f74065d3898 [contrail-topology] [INFO]: Consistent scheduler zk start 04/20/2026 05:25:24.740 7f74002ffd48 [contrail-topology] [ERROR]: Cannot write http_port 5921 to /tmp/contrail-topology.0.http_port 04/20/2026 05:25:24.740 7f74002ffd48 [contrail-topology] [ERROR]: Starting Introspect on HTTP Port 5921 04/20/2026 05:25:24.740 7f73fd82a848 [contrail-topology] [INFO]: Processing event[EvStart] in state[none] 04/20/2026 05:25:24.740 7f73fd82a848 [contrail-topology] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 04/20/2026 05:25:24.740 7f73fd82a848 [contrail-topology] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 04/20/2026 05:25:24.741 7f73fd82a848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/20/2026 05:25:24.741 7f73fd82a848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5649-2. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.50:2181, 10.0.0.33:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Zookeeper:Zookeeper[Connection to Zookeeper initialized] connection down >>, ] >> 04/20/2026 05:25:24.741 7f73fd82a848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/20/2026 05:25:24.741 7f73fd82a848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/20/2026 05:25:24.741 7f73fd82a848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5649-2. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.50:2181, 10.0.0.33:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[Connection to Zookeeper initialized], Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/20/2026 05:25:24.741 7f73fd82a848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/20/2026 05:25:24.741 7f73fd82a848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/20/2026 05:25:24.741 7f73fd82a848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5649-2.:AnalyticsSNMP:contrail-topology:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5921 start_time = 1776662724740749 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.33:8086, 10.0.0.38:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 04/20/2026 05:25:24.741 7f73fd82a848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/20/2026 05:25:24.750 7f73fd7a3848 [contrail-topology] [INFO]: Consistent scheduler listen CONNECTED 04/20/2026 05:25:24.750 7f73fd82a848 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/20/2026 05:25:24.751 7f73fd82a848 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5649-2. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.50:2181, 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/20/2026 05:25:24.751 7f73fd82a848 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/20/2026 05:25:24.886 7f73fd7a3b48 [contrail-topology] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//', 'pyamqp://guest:guest@10.0.0.33:5673//'] 04/20/2026 05:25:24.890 7f73fd7a3b48 [contrail-topology] [WARNING]: RabbitMQ drainer connection down 04/20/2026 05:25:24.891 7f73fd7a3e48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/20/2026 05:25:24.891 7f73fd7a3e48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5649-2. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.50:2181, 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.50:5673, 10.0.0.33:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/20/2026 05:25:24.891 7f73fd7a3e48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/20/2026 05:25:24.891 7f73fd7a3e48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/20/2026 05:25:24.891 7f73fd7a3e48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5649-2. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.50:2181, 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.50:5673, 10.0.0.33:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/20/2026 05:25:24.891 7f73fd7a3e48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/20/2026 05:25:24.919 7f73fd7a3a48 [contrail-topology] [DEBUG]: Waiting for allocation... 04/20/2026 05:25:28.741 7f73fd7a3d48 [contrail-topology] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/20/2026 05:25:28.742 7f73fd7a3d48 [contrail-topology] [INFO]: Session Event: TCP Connected 04/20/2026 05:25:28.742 7f73fd7a3d48 [contrail-topology] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/20/2026 05:25:28.742 7f73fd7a3d48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/20/2026 05:25:28.742 7f73fd7a3d48 [contrail-topology] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5649-2. process_status = [ << module_id = contrail-topology instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.50:2181, 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.50:5673, 10.0.0.33:5673, ] status = Down description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:RabbitMQ[] connection down >>, ] >> 04/20/2026 05:25:28.743 7f73fd7a3d48 [contrail-topology] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/20/2026 05:25:28.743 7f73fd7a3d48 [contrail-topology] [INFO]: Processing event[EvTcpConnected] in state[Connect] 04/20/2026 05:25:28.743 7f73fd7a3d48 [contrail-topology] [DEBUG]: Send sandesh control message. uve type count # 5 04/20/2026 05:25:28.743 7f73fd7a3d48 [contrail-topology] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 04/20/2026 05:25:28.743 7f73fd7a3d48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 04/20/2026 05:25:28.743 7f73fd7a3d48 [contrail-topology] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 04/20/2026 05:25:28.743 7f73fd7a3e48 [contrail-topology] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 04/20/2026 05:25:28.744 7f73fd7a3e48 [contrail-topology] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 04/20/2026 05:25:28.747 7f73fd7a3c48 [contrail-topology] [DEBUG]: Extracted sandesh length: 0000001017 04/20/2026 05:25:28.748 7f73fd7a3c48 [contrail-topology] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 04/20/2026 05:25:28.748 7f73fd7a3e48 [contrail-topology] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 04/20/2026 05:25:28.748 7f73fd7a3e48 [contrail-topology] [DEBUG]: Number of uve types in sandesh control message is 0 04/20/2026 05:25:28.748 7f73fd7a3e48 [contrail-topology] [DEBUG]: send sync_uve in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map 04/20/2026 05:25:28.749 7f73fd7a3e48 [contrail-topology] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 04/20/2026 05:25:28.749 7f73fd7a3e48 [contrail-topology] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 04/20/2026 05:25:28.750 7f73fd7a3d48 [contrail-topology] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 04/20/2026 05:25:28.753 7f73fd7a3d48 [contrail-topology] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 04/20/2026 05:25:30.916 7f73fd7a3b48 [contrail-topology] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/20/2026 05:25:30.925 7f73fd7a3b48 [contrail-topology] [WARNING]: RabbitMQ publish connection down 04/20/2026 05:25:30.937 7f73fd7a3b48 [contrail-topology] [WARNING]: RabbitMQ publish connection ESTABLISHED 04/20/2026 05:25:30.940 7f73fd7a3b48 [contrail-topology] [WARNING]: VNCCassandra started with driver cql 04/20/2026 05:25:30.941 7f73fd7a3b48 [contrail-topology] [WARNING]: initializing cassandra cluster 04/20/2026 05:25:30.942 7f73fd7a3b48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster before create 04/20/2026 05:25:30.946 7f73fd7a3e48 [contrail-topology] [WARNING]: RabbitMQ publish connection down 04/20/2026 05:25:30.948 7f73f778ba48 [contrail-topology] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 04/20/2026 05:25:30.950 7f73f778ba48 [contrail-topology] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 04/20/2026 05:25:30.958 7f73fd7a3b48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster created 04/20/2026 05:25:30.960 7f73fd7a3e48 [contrail-topology] [WARNING]: RabbitMQ publish connection ESTABLISHED 04/20/2026 05:25:30.991 7f73fd7a3b48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 04/20/2026 05:25:31.002 7f73fd7a3b48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 04/20/2026 05:25:31.021 7f73fd7a3b48 [contrail-topology] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 04/20/2026 05:25:33.157 7f73fd7a3948 [contrail-topology] [WARNING]: RabbitMQ drainer connection down 04/20/2026 05:25:55.989 7f73fd7a3a48 [contrail-topology] [DEBUG]: Waiting for allocation... 04/20/2026 05:26:24.849 7f73fd82a848 [contrail-topology] [INFO]: members: ['cn-jenkins-deploy-platform-ansible-os-5649-1.-1', 'cn-jenkins-deploy-platform-ansible-os-5649-2.-1', 'cn-jenkins-deploy-platform-ansible-os-5649-3.-1'] 04/20/2026 05:26:24.849 7f73fd82a848 [contrail-topology] [INFO]: partitions: ['0', '3', '5', '7', '10', '15', '19', '20', '21', '22', '25', '31', '35', '37', '39', '41', '43', '45'] 04/20/2026 05:26:26.027 7f73fd7a3a48 [contrail-topology] [DEBUG]: got work: ['0', '3', '5', '7', '10', '15', '19', '20', '21', '22', '25', '31', '35', '37', '39', '41', '43', '45'] 04/20/2026 05:26:26.027 7f73fd7a3a48 [contrail-topology] [DEBUG]: @populate_work_items(): done! 04/20/2026 05:26:26.027 7f73fd7a3a48 [contrail-topology] [DEBUG]: work items: [] from the list [] 04/20/2026 05:26:26.027 7f73fd7a3a48 [contrail-topology] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:TopologyUVE] map 04/20/2026 05:30:37.489 7f73fd7a3948 [contrail-topology] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/20/2026 05:36:26.054 7f73fd7a3a48 [contrail-topology] [DEBUG]: got work: ['0', '3', '5', '7', '10', '15', '19', '20', '21', '22', '25', '31', '35', '37', '39', '41', '43', '45'] 04/20/2026 05:36:26.054 7f73fd7a3a48 [contrail-topology] [DEBUG]: @populate_work_items(): done! 04/20/2026 05:36:26.055 7f73fd7a3a48 [contrail-topology] [DEBUG]: work items: [] from the list [] 04/20/2026 05:46:02.316 7f73fd82a848 [contrail-topology] [DEBUG]: send sync_uve in the [ObjectAnalyticsSNMPInfo:NodeStatusUVE] map