07/10/2025 01:46:14.527 7f93114fe780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/10/2025 01:46:14.527 7f93114fe780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 111 start_time = 1752111974527055 collector_name = collector_ip = collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 07/10/2025 01:46:14.750 7f931a044c08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.129:5673//'] 07/10/2025 01:46:14.761 7f931a044c08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/10/2025 01:46:14.761 7f9310e63588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.129:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/10/2025 01:46:14.762 7f9310e63588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.129:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/10/2025 01:46:14.778 7f931a044c08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/10/2025 01:46:14.779 7f9310e82cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.129:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/10/2025 01:46:14.823 7f931a044c08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 07/10/2025 01:46:14.830 7f9310e63b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.129:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/10/2025 01:46:14.873 7f931a044c08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/10/2025 01:46:14.875 7f931d221eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.129:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/10/2025 01:46:14.921 7f931a044c08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 07/10/2025 01:46:14.921 7f931a044c08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 07/10/2025 01:46:14.921 7f931a044c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 07/10/2025 01:46:14.922 7f9310e82cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.129:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.129:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/10/2025 01:46:14.925 7f931a044c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 07/10/2025 01:46:15.080 7f931a044c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 07/10/2025 01:46:15.082 7f93113fc588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.129:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.129:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.129:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 07/10/2025 01:46:15.100 7f931a044c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 07/10/2025 01:46:15.101 7f93113d88d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.129:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.129:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.129:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/10/2025 01:46:15.143 7f931a044c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 07/10/2025 01:46:15.220 7f931a044c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 07/10/2025 01:46:15.221 7f9311400390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.129:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.129:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.129:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/10/2025 01:46:15.230 7f93112cc588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.129:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.129:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.129:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 07/10/2025 01:46:15.267 7f93112ce438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.129:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.129:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.129:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/10/2025 01:46:15.806 7f931a044c08 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))",) 07/10/2025 01:46:16.317 7f931a044c08 [contrail-device-manager] [WARNING]: Process 111 prepared to run in Full mode... 07/10/2025 01:46:16.317 7f931a044c08 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 07/10/2025 01:46:16.866 7f931128cac8 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 07/10/2025 01:46:16.867 7f931128cac8 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 07/10/2025 01:46:16.868 7f9311078eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.129:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.129:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.129:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.129,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 07/10/2025 01:46:17.010 7f9311067d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.129:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.129:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.129:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.129,:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/10/2025 01:46:18.528 7f930b71a630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.129:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.129:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.129:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.129,:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/10/2025 01:47:30.353 7f931a044c08 [contrail-device-manager] [WARNING]: NoIdError: default-domain:admin:master-LR 07/10/2025 01:47:30.501 7f931a044c08 [contrail-device-manager] [WARNING]: NoIdError: show_interfaces_template 07/10/2025 01:47:30.548 7f931a044c08 [contrail-device-manager] [WARNING]: NoIdError: show_config_interfaces_template 07/10/2025 01:47:30.598 7f931a044c08 [contrail-device-manager] [WARNING]: NoIdError: show_interfaces_by_names_template 07/10/2025 01:47:30.598 7f931a044c08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 07/10/2025 01:47:30.599 7f931a044c08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 07/10/2025 01:47:30.599 7f931a044c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 07/10/2025 01:47:30.601 7f931a044c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 07/10/2025 01:47:30.651 7f931a044c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 07/10/2025 01:47:30.680 7f931a044c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 07/10/2025 01:47:30.726 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/10/2025 01:47:30.820 7f931a044c08 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 07/10/2025 01:47:30.826 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/10/2025 01:47:30.848 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/10/2025 01:47:30.917 7f931a044c08 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 07/10/2025 01:47:30.920 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/10/2025 01:47:30.920 7f931a044c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 07/10/2025 01:47:31.078 7f931a044c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 07/10/2025 01:47:31.088 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/10/2025 01:47:31.339 7f931a044c08 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 07/10/2025 01:47:31.347 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/10/2025 01:47:31.360 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/10/2025 01:47:31.521 7f931a044c08 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 07/10/2025 01:47:31.528 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/10/2025 01:47:31.549 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/10/2025 01:47:31.733 7f931a044c08 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 07/10/2025 01:47:31.736 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/10/2025 01:47:31.742 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/10/2025 01:47:31.880 7f931a044c08 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 07/10/2025 01:47:31.888 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/10/2025 01:47:31.902 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/10/2025 01:47:32.091 7f931a044c08 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 07/10/2025 01:47:32.097 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/10/2025 01:47:32.103 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/10/2025 01:47:32.208 7f931a044c08 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 07/10/2025 01:47:32.387 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/10/2025 01:47:32.467 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/10/2025 01:47:34.121 7f931a044c08 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 07/10/2025 01:47:34.127 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/10/2025 01:47:34.131 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/10/2025 01:47:34.239 7f931a044c08 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 07/10/2025 01:47:34.254 7f931a044c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/10/2025 01:47:34.305 7f931a044c08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.129:5673//'] 07/10/2025 01:47:34.306 7f931a044c08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/10/2025 01:47:34.329 7f931a044c08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/10/2025 01:47:34.366 7f931a044c08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 07/10/2025 01:47:34.408 7f931a044c08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/10/2025 01:47:34.418 7f9310cd9978 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 07/10/2025 01:47:34.453 7f9310cd9978 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/10/2025 01:48:15.827 7f93114aa978 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 07/10/2025 01:48:15.827 7f93114aa978 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/10/2025 01:48:15.856 7f93114aa978 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/10/2025 01:48:15.978 7f93114aa978 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 07/10/2025 01:48:15.978 7f93114aa978 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/10/2025 01:48:16.009 7f93114aa978 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/10/2025 01:48:16.751 7f93114aa978 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 07/10/2025 01:48:16.752 7f93114aa978 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/10/2025 01:48:16.765 7f93114aa978 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/10/2025 01:48:17.105 7f93114aa978 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 07/10/2025 01:48:17.106 7f93114aa978 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/10/2025 01:48:17.140 7f93114aa978 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED