09/07/2026 01:42:24.889 7f02f0a11180 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/07/2026 01:42:24.890 7f02f0a11180 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1788745344889412 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/07/2026 01:42:24.979 7f02f56e42c0 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.50:5673//'] 09/07/2026 01:42:24.987 7f02f56e42c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 09/07/2026 01:42:24.988 7f02f0a11860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/07/2026 01:42:24.989 7f02f0a11860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/07/2026 01:42:25.006 7f02f56e42c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/07/2026 01:42:25.007 7f02f0a11a40 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/07/2026 01:42:25.025 7f02f56e42c0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 09/07/2026 01:42:25.026 7f02f0c7a360 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/07/2026 01:42:25.037 7f02f56e42c0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/07/2026 01:42:25.038 7f02f0c7a360 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/07/2026 01:42:25.058 7f02f56e42c0 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 09/07/2026 01:42:25.058 7f02f56e42c0 [contrail-device-manager] [WARNING]: initializing cassandra cluster 09/07/2026 01:42:25.058 7f02f56e42c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/07/2026 01:42:25.059 7f02f0c7ab80 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/07/2026 01:42:25.060 7f02f56e42c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/07/2026 01:42:25.103 7f02f56e42c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/07/2026 01:42:25.105 7f02f0a9d7c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/07/2026 01:42:25.117 7f02f56e42c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 09/07/2026 01:42:25.118 7f02f0a9d720 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/07/2026 01:42:25.124 7f02f56e42c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 09/07/2026 01:42:25.139 7f02f56e42c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 09/07/2026 01:42:25.386 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/07/2026 01:42:25.597 7f02f56e42c0 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 09/07/2026 01:42:25.602 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/07/2026 01:42:26.081 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/07/2026 01:42:26.177 7f02f56e42c0 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 09/07/2026 01:42:26.184 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/07/2026 01:42:26.184 7f02f0a9dc20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/07/2026 01:42:26.208 7f02f0c7aa40 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 09/07/2026 01:42:26.216 7f02f0c7aa40 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/07/2026 01:42:26.722 7f02f56e42c0 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))") 09/07/2026 01:42:27.224 7f02f56e42c0 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 09/07/2026 01:42:27.224 7f02f56e42c0 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 09/07/2026 01:42:27.607 7f02f0cfc540 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 09/07/2026 01:42:27.607 7f02f0cfc540 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 09/07/2026 01:42:27.608 7f02ebd065e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 09/07/2026 01:42:27.617 7f02ebd06c20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 09/07/2026 01:42:28.896 7f02ebd06900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-430-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50,:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 09/07/2026 01:42:30.641 7f02f56e42c0 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 09/07/2026 01:42:30.641 7f02f56e42c0 [contrail-device-manager] [ERROR]: Unable to load init data 09/07/2026 01:42:30.641 7f02f56e42c0 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 09/07/2026 01:42:30.641 7f02f56e42c0 [contrail-device-manager] [WARNING]: initializing cassandra cluster 09/07/2026 01:42:30.644 7f02f56e42c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/07/2026 01:42:30.648 7f02f56e42c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/07/2026 01:42:30.717 7f02f56e42c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/07/2026 01:42:30.733 7f02f56e42c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 09/07/2026 01:42:30.751 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/07/2026 01:42:30.803 7f02f56e42c0 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 09/07/2026 01:42:30.805 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/07/2026 01:42:30.811 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/07/2026 01:42:30.857 7f02f56e42c0 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 09/07/2026 01:42:30.862 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/07/2026 01:42:30.862 7f02f56e42c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 09/07/2026 01:42:30.891 7f02f56e42c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 09/07/2026 01:42:30.896 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/07/2026 01:42:31.026 7f02f56e42c0 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 09/07/2026 01:42:31.033 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/07/2026 01:42:31.049 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/07/2026 01:42:31.221 7f02f56e42c0 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 09/07/2026 01:42:31.225 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/07/2026 01:42:31.233 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/07/2026 01:42:31.383 7f02f56e42c0 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 09/07/2026 01:42:31.386 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/07/2026 01:42:31.391 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/07/2026 01:42:31.489 7f02f56e42c0 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 09/07/2026 01:42:31.500 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/07/2026 01:42:31.505 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/07/2026 01:42:31.642 7f02f56e42c0 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 09/07/2026 01:42:31.644 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/07/2026 01:42:31.651 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/07/2026 01:42:31.748 7f02f56e42c0 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 09/07/2026 01:42:31.756 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/07/2026 01:42:31.764 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/07/2026 01:42:31.903 7f02f56e42c0 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 09/07/2026 01:42:31.906 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/07/2026 01:42:31.913 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/07/2026 01:42:32.018 7f02f56e42c0 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 09/07/2026 01:42:32.021 7f02f56e42c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/07/2026 01:42:32.049 7f02f56e42c0 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.50:5673//'] 09/07/2026 01:42:32.049 7f02f56e42c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 09/07/2026 01:42:32.058 7f02f56e42c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/07/2026 01:42:32.070 7f02f56e42c0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 09/07/2026 01:42:32.084 7f02f56e42c0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/07/2026 01:42:32.086 7f02ebd3b860 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 09/07/2026 01:42:32.109 7f02ebd3b860 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED