12/08/2025 01:42:05.522 7f7a573a3ea0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-277-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/08/2025 01:42:05.522 7f7a573a3ea0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-277-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1765158125522416 collector_name = collector_ip = collector_list = [ 10.0.0.19:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 12/08/2025 01:42:05.554 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.19:5673//'] 12/08/2025 01:42:05.564 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 12/08/2025 01:42:05.564 7f7a576807c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-277-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 12/08/2025 01:42:05.565 7f7a576807c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-277-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 12/08/2025 01:42:05.584 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 12/08/2025 01:42:05.584 7f7a57680900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-277-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/08/2025 01:42:05.617 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 12/08/2025 01:42:05.618 7f7a573420e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-277-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 12/08/2025 01:42:05.630 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 12/08/2025 01:42:05.630 7f7a573420e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-277-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/08/2025 01:42:05.664 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 12/08/2025 01:42:05.664 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: initializing cassandra cluster 12/08/2025 01:42:05.665 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 12/08/2025 01:42:05.665 7f7a57342900 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-277-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 12/08/2025 01:42:05.670 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 12/08/2025 01:42:05.933 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 12/08/2025 01:42:05.935 7f7a57680540 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-277-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.19:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 12/08/2025 01:42:05.953 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 12/08/2025 01:42:05.953 7f7a57342220 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-277-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.19:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 12/08/2025 01:42:05.973 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 12/08/2025 01:42:06.021 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 12/08/2025 01:42:06.968 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/08/2025 01:42:06.973 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 12/08/2025 01:42:06.977 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/08/2025 01:42:07.327 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/08/2025 01:42:07.523 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 12/08/2025 01:42:07.535 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/08/2025 01:42:08.004 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/08/2025 01:42:08.012 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 12/08/2025 01:42:08.017 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/08/2025 01:42:08.554 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/08/2025 01:42:08.643 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 12/08/2025 01:42:08.647 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/08/2025 01:42:08.648 7f7a576807c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-277-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.19:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/08/2025 01:42:08.653 7f7a573427c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-277-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.19:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 12/08/2025 01:42:08.660 7f7a573427c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-277-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.19:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/08/2025 01:42:09.169 7f7a5bbb2e00 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))") 12/08/2025 01:42:09.527 7f7a573427c0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-277-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.19:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.19:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.19:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.19:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/08/2025 01:42:09.671 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 12/08/2025 01:42:09.671 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 12/08/2025 01:42:09.766 7f7a57625220 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 12/08/2025 01:42:09.767 7f7a57625220 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 12/08/2025 01:42:13.254 7f7a5bbb2e00 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 12/08/2025 01:42:13.254 7f7a5bbb2e00 [contrail-device-manager] [ERROR]: Unable to load init data 12/08/2025 01:42:13.254 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 12/08/2025 01:42:13.254 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: initializing cassandra cluster 12/08/2025 01:42:13.254 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 12/08/2025 01:42:13.266 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 12/08/2025 01:42:13.339 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 12/08/2025 01:42:13.417 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 12/08/2025 01:42:13.470 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/08/2025 01:42:13.549 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 12/08/2025 01:42:13.557 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/08/2025 01:42:13.578 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/08/2025 01:42:13.696 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 12/08/2025 01:42:13.712 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/08/2025 01:42:13.712 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 12/08/2025 01:42:13.848 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 12/08/2025 01:42:13.860 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/08/2025 01:42:14.023 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 12/08/2025 01:42:14.028 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/08/2025 01:42:14.040 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/08/2025 01:42:14.260 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 12/08/2025 01:42:14.273 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/08/2025 01:42:14.287 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/08/2025 01:42:14.540 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 12/08/2025 01:42:14.550 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/08/2025 01:42:14.746 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/08/2025 01:42:14.986 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 12/08/2025 01:42:14.996 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/08/2025 01:42:15.175 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/08/2025 01:42:15.390 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 12/08/2025 01:42:15.413 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/08/2025 01:42:15.821 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/08/2025 01:42:15.910 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 12/08/2025 01:42:15.917 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/08/2025 01:42:16.141 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/08/2025 01:42:16.440 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 12/08/2025 01:42:16.451 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/08/2025 01:42:16.733 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/08/2025 01:42:16.808 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 12/08/2025 01:42:16.817 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/08/2025 01:42:16.854 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.19:5673//'] 12/08/2025 01:42:16.855 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 12/08/2025 01:42:16.872 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 12/08/2025 01:42:16.883 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 12/08/2025 01:42:16.899 7f7a5bbb2e00 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 12/08/2025 01:42:16.910 7f7a56c0c180 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 12/08/2025 01:42:16.945 7f7a56c0c180 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 12/08/2025 01:44:49.840 7f7a563040e0 [contrail-device-manager] [WARNING]: Inconsistency discovered. wanted=1, received=0, missing={'803ba9b8-46ec-4ac7-afa8-e7fe8035ed1e'}. This may indicate that the cluster needs a 'nodetool repair'. 12/08/2025 01:44:49.840 7f7a563040e0 [contrail-device-manager] [WARNING]: instance_ip uuid 803ba9b8-46ec-4ac7-afa8-e7fe8035ed1e update caused NoIdError