09/15/2025 01:36:04.550 7f8c623ac0e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/15/2025 01:36:04.550 7f8c623ac0e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 107 start_time = 1757900164549846 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/15/2025 01:36:04.682 7f8c67764100 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 09/15/2025 01:36:04.697 7f8c67764100 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 09/15/2025 01:36:04.698 7f8c623ac9a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/15/2025 01:36:04.698 7f8c623ac9a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/15/2025 01:36:04.746 7f8c67764100 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/15/2025 01:36:04.746 7f8c623acae0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/15/2025 01:36:04.811 7f8c67764100 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 09/15/2025 01:36:04.812 7f8c626c92c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/15/2025 01:36:04.834 7f8c67764100 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/15/2025 01:36:04.835 7f8c626c92c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/15/2025 01:36:04.881 7f8c67764100 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 09/15/2025 01:36:04.881 7f8c67764100 [contrail-device-manager] [WARNING]: initializing cassandra cluster 09/15/2025 01:36:04.881 7f8c67764100 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/15/2025 01:36:04.883 7f8c626c9ae0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/15/2025 01:36:04.884 7f8c67764100 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/15/2025 01:36:05.231 7f8c67764100 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/15/2025 01:36:05.233 7f8c623b5400 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/15/2025 01:36:05.248 7f8c67764100 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 09/15/2025 01:36:05.255 7f8c623b5360 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/15/2025 01:36:05.391 7f8c67764100 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 09/15/2025 01:36:05.546 7f8c67764100 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 09/15/2025 01:36:05.547 7f8c6277c0e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/15/2025 01:36:05.554 7f8c626c99a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 09/15/2025 01:36:05.579 7f8c626c99a0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/15/2025 01:36:06.083 7f8c67764100 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))") 09/15/2025 01:36:06.596 7f8c67764100 [contrail-device-manager] [WARNING]: Process 107 prepared to run in Full mode... 09/15/2025 01:36:06.596 7f8c67764100 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 09/15/2025 01:36:06.693 7f8c62576360 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 09/15/2025 01:36:06.694 7f8c62576360 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 09/15/2025 01:36:07.266 7f8c615ff040 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 09/15/2025 01:36:07.506 7f8c615ff400 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22,:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 09/15/2025 01:36:08.552 7f8c615ff360 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-249-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22,:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 09/15/2025 01:36:13.563 7f8c67764100 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 09/15/2025 01:36:13.563 7f8c67764100 [contrail-device-manager] [ERROR]: Unable to load init data 09/15/2025 01:36:13.563 7f8c67764100 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 09/15/2025 01:36:13.563 7f8c67764100 [contrail-device-manager] [WARNING]: initializing cassandra cluster 09/15/2025 01:36:13.563 7f8c67764100 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/15/2025 01:36:13.571 7f8c67764100 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/15/2025 01:36:13.759 7f8c67764100 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/15/2025 01:36:13.800 7f8c67764100 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 09/15/2025 01:36:13.930 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/15/2025 01:36:14.118 7f8c67764100 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 09/15/2025 01:36:14.171 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/15/2025 01:36:14.244 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/15/2025 01:36:14.613 7f8c67764100 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 09/15/2025 01:36:14.620 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/15/2025 01:36:14.620 7f8c67764100 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 09/15/2025 01:36:14.874 7f8c67764100 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 09/15/2025 01:36:14.899 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/15/2025 01:36:15.435 7f8c67764100 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 09/15/2025 01:36:15.455 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/15/2025 01:36:15.523 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/15/2025 01:36:16.339 7f8c67764100 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 09/15/2025 01:36:16.351 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/15/2025 01:36:16.376 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/15/2025 01:36:16.775 7f8c67764100 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 09/15/2025 01:36:16.800 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/15/2025 01:36:16.862 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/15/2025 01:36:17.176 7f8c67764100 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 09/15/2025 01:36:17.183 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/15/2025 01:36:17.203 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/15/2025 01:36:17.419 7f8c67764100 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 09/15/2025 01:36:17.446 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/15/2025 01:36:17.529 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/15/2025 01:36:17.972 7f8c67764100 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 09/15/2025 01:36:17.984 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/15/2025 01:36:18.019 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/15/2025 01:36:18.516 7f8c67764100 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 09/15/2025 01:36:18.575 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/15/2025 01:36:18.591 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/15/2025 01:36:18.896 7f8c67764100 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 09/15/2025 01:36:18.920 7f8c67764100 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/15/2025 01:36:18.967 7f8c67764100 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 09/15/2025 01:36:18.968 7f8c67764100 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 09/15/2025 01:36:19.024 7f8c67764100 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/15/2025 01:36:19.073 7f8c67764100 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 09/15/2025 01:36:19.113 7f8c67764100 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/15/2025 01:36:19.117 7f8c61aac720 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 09/15/2025 01:36:19.158 7f8c61aac720 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/15/2025 01:36:20.029 7f8c615ff360 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 09/15/2025 01:36:20.029 7f8c615ff360 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 09/15/2025 01:36:20.106 7f8c615ff360 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED