07/07/2025 01:43:25.150 7f24a2560780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/07/2025 01:43:25.151 7f24a2560780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 start_time = 1751852605150669 collector_name = collector_ip = collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 07/07/2025 01:43:25.476 7f24ab060288 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.56:5673//'] 07/07/2025 01:43:25.602 7f24ab060288 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/07/2025 01:43:25.603 7f24a1f12588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/07/2025 01:43:25.603 7f24a1f12588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/07/2025 01:43:25.628 7f24ab060288 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/07/2025 01:43:25.633 7f24a1f23eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/07/2025 01:43:25.704 7f24ab060288 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 07/07/2025 01:43:25.706 7f24a1f12b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/07/2025 01:43:25.727 7f24ab060288 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/07/2025 01:43:25.728 7f24ae28feb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/07/2025 01:43:25.775 7f24ab060288 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 07/07/2025 01:43:25.775 7f24ab060288 [contrail-device-manager] [WARNING]: initializing cassandra cluster 07/07/2025 01:43:25.775 7f24ab060288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 07/07/2025 01:43:25.776 7f24a1f23eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/07/2025 01:43:25.777 7f24ab060288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 07/07/2025 01:43:26.019 7f24ab060288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 07/07/2025 01:43:26.020 7f24a2461978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 07/07/2025 01:43:26.029 7f24ab060288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 07/07/2025 01:43:26.030 7f24a24a9780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/07/2025 01:43:26.086 7f24ab060288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 07/07/2025 01:43:26.308 7f24ab060288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 07/07/2025 01:43:26.309 7f24a2438cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/07/2025 01:43:26.425 7f24a2468630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 07/07/2025 01:43:26.554 7f24a246c4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/07/2025 01:43:27.095 7f24ab060288 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))",) 07/07/2025 01:43:27.603 7f24ab060288 [contrail-device-manager] [WARNING]: Process 114 prepared to run in Full mode... 07/07/2025 01:43:27.604 7f24ab060288 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 07/07/2025 01:43:28.033 7f24a2362390 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 07/07/2025 01:43:28.033 7f24a2362390 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 07/07/2025 01:43:28.034 7f249b79f0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.56,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 07/07/2025 01:43:28.043 7f24a1f5ae10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.56,:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 07/07/2025 01:43:29.155 7f24a2250588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-225-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.56,:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 07/07/2025 01:44:41.154 7f24ab060288 [contrail-device-manager] [WARNING]: NoIdError: default-domain:admin:master-LR 07/07/2025 01:44:41.291 7f24ab060288 [contrail-device-manager] [WARNING]: NoIdError: show_interfaces_template 07/07/2025 01:44:41.336 7f24ab060288 [contrail-device-manager] [WARNING]: NoIdError: show_config_interfaces_template 07/07/2025 01:44:41.381 7f24ab060288 [contrail-device-manager] [WARNING]: NoIdError: show_interfaces_by_names_template 07/07/2025 01:44:41.382 7f24ab060288 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 07/07/2025 01:44:41.382 7f24ab060288 [contrail-device-manager] [WARNING]: initializing cassandra cluster 07/07/2025 01:44:41.382 7f24ab060288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 07/07/2025 01:44:41.384 7f24ab060288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 07/07/2025 01:44:41.405 7f24ab060288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 07/07/2025 01:44:41.411 7f24ab060288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 07/07/2025 01:44:41.421 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/07/2025 01:44:41.459 7f24ab060288 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 07/07/2025 01:44:41.462 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/07/2025 01:44:41.468 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/07/2025 01:44:41.530 7f24ab060288 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 07/07/2025 01:44:41.536 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/07/2025 01:44:41.536 7f24ab060288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 07/07/2025 01:44:41.557 7f24ab060288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 07/07/2025 01:44:41.563 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/07/2025 01:44:41.656 7f24ab060288 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 07/07/2025 01:44:41.659 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/07/2025 01:44:41.663 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/07/2025 01:44:41.722 7f24ab060288 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 07/07/2025 01:44:41.725 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/07/2025 01:44:41.737 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/07/2025 01:44:41.818 7f24ab060288 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 07/07/2025 01:44:41.821 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/07/2025 01:44:41.826 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/07/2025 01:44:41.881 7f24ab060288 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 07/07/2025 01:44:41.885 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/07/2025 01:44:41.890 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/07/2025 01:44:41.972 7f24ab060288 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 07/07/2025 01:44:41.975 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/07/2025 01:44:41.985 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/07/2025 01:44:42.032 7f24ab060288 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 07/07/2025 01:44:42.034 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/07/2025 01:44:42.038 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/07/2025 01:44:42.115 7f24ab060288 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 07/07/2025 01:44:42.122 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/07/2025 01:44:42.127 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/07/2025 01:44:42.183 7f24ab060288 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 07/07/2025 01:44:42.186 7f24ab060288 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/07/2025 01:44:42.212 7f24ab060288 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.56:5673//'] 07/07/2025 01:44:42.212 7f24ab060288 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/07/2025 01:44:42.218 7f24ab060288 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/07/2025 01:44:42.224 7f24ab060288 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 07/07/2025 01:44:42.229 7f24ab060288 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/07/2025 01:44:42.233 7f24a1d1c0f0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 07/07/2025 01:44:42.242 7f24a1d1c0f0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/07/2025 01:45:45.481 7f24a1d19c18 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 07/07/2025 01:45:45.481 7f24a1d19c18 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/07/2025 01:45:45.503 7f24a1d19c18 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/07/2025 01:45:45.641 7f24a1d19c18 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 07/07/2025 01:45:45.641 7f24a1d19c18 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/07/2025 01:45:45.660 7f24a1d19c18 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/07/2025 01:45:46.328 7f24a1d19c18 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 07/07/2025 01:45:46.328 7f24a1d19c18 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/07/2025 01:45:46.341 7f24a1d19c18 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED