01/12/2026 01:33:21.876 7f96acb5b040 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-287-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/12/2026 01:33:21.876 7f96acb5b040 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-287-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1768181601875757 collector_name = collector_ip = collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 01/12/2026 01:33:21.907 7f96b11791c0 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.33:5673//'] 01/12/2026 01:33:21.916 7f96b11791c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 01/12/2026 01:33:21.917 7f96acb5b900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-287-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 01/12/2026 01:33:21.917 7f96acb5b900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-287-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 01/12/2026 01:33:21.930 7f96b11791c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 01/12/2026 01:33:21.930 7f96acb5ba40 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-287-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/12/2026 01:33:21.947 7f96b11791c0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 01/12/2026 01:33:21.948 7f96ac90c220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-287-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 01/12/2026 01:33:21.956 7f96b11791c0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 01/12/2026 01:33:21.956 7f96ac90c220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-287-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/12/2026 01:33:22.003 7f96b11791c0 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 01/12/2026 01:33:22.003 7f96b11791c0 [contrail-device-manager] [WARNING]: initializing cassandra cluster 01/12/2026 01:33:22.004 7f96b11791c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 01/12/2026 01:33:22.006 7f96ac90ca40 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-287-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 01/12/2026 01:33:22.007 7f96b11791c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 01/12/2026 01:33:22.443 7f96b11791c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 01/12/2026 01:33:22.446 7f96ac905860 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-287-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 01/12/2026 01:33:22.471 7f96b11791c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 01/12/2026 01:33:22.474 7f96ac9057c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-287-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 01/12/2026 01:33:22.512 7f96b11791c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 01/12/2026 01:33:22.566 7f96b11791c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 01/12/2026 01:33:23.947 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/12/2026 01:33:23.958 7f96b11791c0 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 01/12/2026 01:33:24.044 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/12/2026 01:33:24.323 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/12/2026 01:33:24.928 7f96b11791c0 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 01/12/2026 01:33:24.996 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/12/2026 01:33:24.997 7f96af115540 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-287-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/12/2026 01:33:25.063 7f96ac90c900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-287-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 01/12/2026 01:33:25.114 7f96ac90c900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-287-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/12/2026 01:33:25.618 7f96b11791c0 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))") 01/12/2026 01:33:25.878 7f96acb5b900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-287-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/12/2026 01:33:26.122 7f96b11791c0 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 01/12/2026 01:33:26.122 7f96b11791c0 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 01/12/2026 01:33:26.269 7f96acbef180 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 01/12/2026 01:33:26.269 7f96acbef180 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 01/12/2026 01:33:30.090 7f96b11791c0 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 01/12/2026 01:33:30.091 7f96b11791c0 [contrail-device-manager] [ERROR]: Unable to load init data 01/12/2026 01:33:30.091 7f96b11791c0 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 01/12/2026 01:33:30.091 7f96b11791c0 [contrail-device-manager] [WARNING]: initializing cassandra cluster 01/12/2026 01:33:30.091 7f96b11791c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 01/12/2026 01:33:30.093 7f96b11791c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 01/12/2026 01:33:30.121 7f96b11791c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 01/12/2026 01:33:30.132 7f96b11791c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 01/12/2026 01:33:30.152 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/12/2026 01:33:30.202 7f96b11791c0 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 01/12/2026 01:33:30.206 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/12/2026 01:33:30.220 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/12/2026 01:33:30.305 7f96b11791c0 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 01/12/2026 01:33:30.310 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/12/2026 01:33:30.310 7f96b11791c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 01/12/2026 01:33:30.367 7f96b11791c0 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 01/12/2026 01:33:30.393 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/12/2026 01:33:30.614 7f96b11791c0 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 01/12/2026 01:33:30.626 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/12/2026 01:33:30.778 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/12/2026 01:33:30.929 7f96b11791c0 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 01/12/2026 01:33:30.935 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/12/2026 01:33:31.115 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/12/2026 01:33:31.293 7f96b11791c0 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 01/12/2026 01:33:31.297 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/12/2026 01:33:31.442 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/12/2026 01:33:31.618 7f96b11791c0 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 01/12/2026 01:33:31.631 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/12/2026 01:33:31.824 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/12/2026 01:33:32.222 7f96b11791c0 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 01/12/2026 01:33:32.241 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/12/2026 01:33:32.432 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/12/2026 01:33:32.721 7f96b11791c0 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 01/12/2026 01:33:32.737 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/12/2026 01:33:33.129 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/12/2026 01:33:33.333 7f96b11791c0 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 01/12/2026 01:33:33.342 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/12/2026 01:33:33.659 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/12/2026 01:33:33.873 7f96b11791c0 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 01/12/2026 01:33:33.883 7f96b11791c0 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/12/2026 01:33:33.916 7f96b11791c0 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.33:5673//'] 01/12/2026 01:33:33.916 7f96b11791c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 01/12/2026 01:33:33.959 7f96b11791c0 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 01/12/2026 01:33:33.997 7f96b11791c0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 01/12/2026 01:33:34.062 7f96b11791c0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 01/12/2026 01:33:34.073 7f96a7fcb400 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 01/12/2026 01:33:34.105 7f96a7fcb400 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED