10/26/2026 01:48:45.436 7fda4908b180 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 10/26/2026 01:48:45.437 7fda4908b180 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1792979325436655 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/26/2026 01:48:45.597 7fda4db031c0 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.50:5673//'] 10/26/2026 01:48:45.622 7fda4db031c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 10/26/2026 01:48:45.623 7fda4908b900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 10/26/2026 01:48:45.623 7fda4908b900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 10/26/2026 01:48:45.636 7fda4db031c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 10/26/2026 01:48:45.636 7fda4908bae0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 10/26/2026 01:48:45.697 7fda4db031c0 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 10/26/2026 01:48:45.697 7fda4db031c0 [contrail-device-manager] [WARNING]: initializing cassandra cluster 10/26/2026 01:48:45.698 7fda4db031c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 10/26/2026 01:48:45.700 7fda48e7ec20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 10/26/2026 01:48:45.702 7fda4db031c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 10/26/2026 01:48:45.742 7fda4db031c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 10/26/2026 01:48:45.746 7fda48ec8c20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 10/26/2026 01:48:45.767 7fda4db031c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 10/26/2026 01:48:45.768 7fda4908b900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 10/26/2026 01:48:45.780 7fda4db031c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 10/26/2026 01:48:45.828 7fda4db031c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 10/26/2026 01:48:45.829 7fda49141860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 10/26/2026 01:48:45.843 7fda48e7eae0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 10/26/2026 01:48:45.867 7fda48e7eae0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 10/26/2026 01:48:46.378 7fda4db031c0 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))") 10/26/2026 01:48:46.893 7fda4db031c0 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 10/26/2026 01:48:46.893 7fda4db031c0 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 10/26/2026 01:48:47.455 7fda491017c0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 10/26/2026 01:48:47.456 7fda491017c0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 10/26/2026 01:48:47.456 7fda48f12a40 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 10/26/2026 01:48:47.467 7fda4db031c0 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 10/26/2026 01:48:47.467 7fda4db031c0 [contrail-device-manager] [ERROR]: Unable to load init data 10/26/2026 01:48:47.467 7fda4db031c0 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 10/26/2026 01:48:47.467 7fda4db031c0 [contrail-device-manager] [WARNING]: initializing cassandra cluster 10/26/2026 01:48:47.468 7fda4db031c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 10/26/2026 01:48:47.469 7fda48f12d60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 10/26/2026 01:48:47.469 7fda48f12d60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 10/26/2026 01:48:47.470 7fda4db031c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 10/26/2026 01:48:47.742 7fda4db031c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 10/26/2026 01:48:47.745 7fda431219a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Initializing description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 10/26/2026 01:48:47.765 7fda4db031c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 10/26/2026 01:48:47.766 7fda43121220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 10/26/2026 01:48:48.272 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 10/26/2026 01:48:48.407 7fda4db031c0 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 10/26/2026 01:48:48.418 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 10/26/2026 01:48:48.934 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 10/26/2026 01:48:49.045 7fda4db031c0 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 10/26/2026 01:48:49.051 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 10/26/2026 01:48:49.051 7fda4db031c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 10/26/2026 01:48:49.130 7fda4db031c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 10/26/2026 01:48:49.438 7fda434e39a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-437-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 10/26/2026 01:48:49.590 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 10/26/2026 01:48:49.854 7fda4db031c0 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 10/26/2026 01:48:49.873 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 10/26/2026 01:48:50.200 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 10/26/2026 01:48:50.389 7fda4db031c0 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 10/26/2026 01:48:50.396 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 10/26/2026 01:48:51.022 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 10/26/2026 01:48:51.158 7fda4db031c0 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 10/26/2026 01:48:51.167 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 10/26/2026 01:48:51.311 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 10/26/2026 01:48:51.426 7fda4db031c0 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 10/26/2026 01:48:51.433 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 10/26/2026 01:48:51.644 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 10/26/2026 01:48:51.898 7fda4db031c0 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 10/26/2026 01:48:51.911 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 10/26/2026 01:48:52.074 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 10/26/2026 01:48:52.289 7fda4db031c0 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 10/26/2026 01:48:52.299 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 10/26/2026 01:48:52.332 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 10/26/2026 01:48:52.536 7fda4db031c0 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 10/26/2026 01:48:52.547 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 10/26/2026 01:48:52.578 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 10/26/2026 01:48:52.704 7fda4db031c0 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 10/26/2026 01:48:52.712 7fda4db031c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 10/26/2026 01:48:52.737 7fda4db031c0 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.50:5673//'] 10/26/2026 01:48:52.737 7fda4db031c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 10/26/2026 01:48:52.757 7fda4db031c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED