05/15/2025 01:40:12.862 7f81fd850780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/15/2025 01:40:12.862 7f81fd850780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 111 start_time = 1747273212862073 collector_name = collector_ip = collector_list = [ 10.0.0.19:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 05/15/2025 01:40:13.076 7f8206362088 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.19:5673//'] 05/15/2025 01:40:13.088 7f8206362088 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 05/15/2025 01:40:13.089 7f81fd1fc588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 05/15/2025 01:40:13.089 7f81fd1fc588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 05/15/2025 01:40:13.131 7f8206362088 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 05/15/2025 01:40:13.150 7f81fd211eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/15/2025 01:40:13.309 7f8206362088 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 05/15/2025 01:40:13.310 7f81fd1fcb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 05/15/2025 01:40:13.317 7f8206362088 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 05/15/2025 01:40:13.317 7f820957ceb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/15/2025 01:40:13.378 7f8206362088 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 05/15/2025 01:40:13.378 7f8206362088 [contrail-device-manager] [WARNING]: initializing cassandra cluster 05/15/2025 01:40:13.380 7f8206362088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 05/15/2025 01:40:13.384 7f81fd211eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 05/15/2025 01:40:13.385 7f8206362088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 05/15/2025 01:40:13.653 7f8206362088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 05/15/2025 01:40:13.655 7f81fd750e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.19:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 05/15/2025 01:40:13.687 7f8206362088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 05/15/2025 01:40:13.687 7f81fd7a2c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.19:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 05/15/2025 01:40:13.749 7f8206362088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 05/15/2025 01:40:13.832 7f8206362088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 05/15/2025 01:40:13.833 7f81fd7526d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.19:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/15/2025 01:40:13.875 7f81fd75f8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.19:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 05/15/2025 01:40:13.910 7f81fd761780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.19:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/15/2025 01:40:14.483 7f8206362088 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))",) 05/15/2025 01:40:15.004 7f8206362088 [contrail-device-manager] [WARNING]: Process 111 prepared to run in Full mode... 05/15/2025 01:40:15.004 7f8206362088 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 05/15/2025 01:40:15.958 7f81fd635a20 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 05/15/2025 01:40:15.959 7f81fd635a20 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 05/15/2025 01:40:15.960 7f81fd4588d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.19:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.19,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 05/15/2025 01:40:16.256 7f81f6adc2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.19:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.19,:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 05/15/2025 01:40:16.875 7f81f7a98240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-188-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.19:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.19:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.19,:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 05/15/2025 01:41:33.308 7f8206362088 [contrail-device-manager] [WARNING]: NoIdError: default-domain:admin:master-LR 05/15/2025 01:41:33.448 7f8206362088 [contrail-device-manager] [WARNING]: NoIdError: show_interfaces_template 05/15/2025 01:41:33.495 7f8206362088 [contrail-device-manager] [WARNING]: NoIdError: show_config_interfaces_template 05/15/2025 01:41:33.542 7f8206362088 [contrail-device-manager] [WARNING]: NoIdError: show_interfaces_by_names_template 05/15/2025 01:41:33.542 7f8206362088 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 05/15/2025 01:41:33.542 7f8206362088 [contrail-device-manager] [WARNING]: initializing cassandra cluster 05/15/2025 01:41:33.543 7f8206362088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 05/15/2025 01:41:33.544 7f8206362088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 05/15/2025 01:41:33.566 7f8206362088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 05/15/2025 01:41:33.577 7f8206362088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 05/15/2025 01:41:33.591 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/15/2025 01:41:33.656 7f8206362088 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 05/15/2025 01:41:33.659 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/15/2025 01:41:33.667 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/15/2025 01:41:33.724 7f8206362088 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 05/15/2025 01:41:33.727 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/15/2025 01:41:33.727 7f8206362088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 05/15/2025 01:41:33.755 7f8206362088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 05/15/2025 01:41:33.760 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/15/2025 01:41:33.859 7f8206362088 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 05/15/2025 01:41:33.862 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/15/2025 01:41:33.869 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/15/2025 01:41:33.937 7f8206362088 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 05/15/2025 01:41:33.945 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/15/2025 01:41:33.952 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/15/2025 01:41:34.047 7f8206362088 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 05/15/2025 01:41:34.053 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/15/2025 01:41:34.057 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/15/2025 01:41:34.121 7f8206362088 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 05/15/2025 01:41:34.126 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/15/2025 01:41:34.135 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/15/2025 01:41:34.227 7f8206362088 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 05/15/2025 01:41:34.231 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/15/2025 01:41:34.241 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/15/2025 01:41:34.304 7f8206362088 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 05/15/2025 01:41:34.307 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/15/2025 01:41:34.316 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/15/2025 01:41:34.449 7f8206362088 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 05/15/2025 01:41:34.452 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/15/2025 01:41:34.459 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 05/15/2025 01:41:34.518 7f8206362088 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 05/15/2025 01:41:34.521 7f8206362088 [contrail-device-manager] [WARNING]: Zookeeper lock released. 05/15/2025 01:41:34.548 7f8206362088 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.19:5673//'] 05/15/2025 01:41:34.549 7f8206362088 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 05/15/2025 01:41:34.561 7f8206362088 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 05/15/2025 01:41:34.574 7f8206362088 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 05/15/2025 01:41:34.579 7f8206362088 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 05/15/2025 01:41:34.581 7f81fd1c5a20 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 05/15/2025 01:41:34.594 7f81fd1c5a20 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 05/15/2025 01:42:24.173 7f81fd1c5438 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 05/15/2025 01:42:24.173 7f81fd1c5438 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 05/15/2025 01:42:24.181 7f81fd1c5438 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 05/15/2025 01:42:24.319 7f81fd1c5438 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 05/15/2025 01:42:24.326 7f81fd1c5438 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 05/15/2025 01:42:24.338 7f81fd1c5438 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 05/15/2025 01:42:25.110 7f81fd1c5438 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 05/15/2025 01:42:25.110 7f81fd1c5438 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 05/15/2025 01:42:25.131 7f81fd1c5438 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 05/15/2025 01:43:25.793 7f81fd1c5438 [contrail-device-manager] [WARNING]: Inconsistency discovered. wanted=1, received=0, missing={'70599d06-ee4d-4b40-91e0-509663649e21'}. This may indicate that the cluster needs a 'nodetool repair'. 05/15/2025 01:43:25.794 7f81fd1c5438 [contrail-device-manager] [WARNING]: instance_ip uuid 70599d06-ee4d-4b40-91e0-509663649e21 update caused NoIdError