09/05/2024 02:02:32.803 7f59a99e5cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/05/2024 02:02:32.803 7f59a99e5cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1725501752802794 collector_name = collector_ip = collector_list = [ 10.0.0.28:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/05/2024 02:02:32.914 7f59b22ef148 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.28:5673//'] 09/05/2024 02:02:32.932 7f59b22ef148 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 09/05/2024 02:02:32.933 7f59a99b58d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.28:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/05/2024 02:02:32.934 7f59a99b58d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.28:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/05/2024 02:02:32.980 7f59b22ef148 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/05/2024 02:02:32.980 7f59a9879cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.28:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/05/2024 02:02:32.997 7f59b22ef148 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 09/05/2024 02:02:32.998 7f59a99b56d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.28:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/05/2024 02:02:33.013 7f59b22ef148 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/05/2024 02:02:33.013 7f59b81fe978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.28:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/05/2024 02:02:33.046 7f59b22ef148 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 09/05/2024 02:02:33.046 7f59b22ef148 [contrail-device-manager] [WARNING]: initializing cassandra cluster 09/05/2024 02:02:33.046 7f59b22ef148 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/05/2024 02:02:33.055 7f59a987cc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.28:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.28:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/05/2024 02:02:33.062 7f59b22ef148 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/05/2024 02:02:33.275 7f59b22ef148 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/05/2024 02:02:33.287 7f59a9741780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.28:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.28:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.28:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/05/2024 02:02:33.311 7f59b22ef148 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 09/05/2024 02:02:33.311 7f59a97d8588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.28:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.28:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.28:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/05/2024 02:02:33.374 7f59b22ef148 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 09/05/2024 02:02:33.523 7f59b22ef148 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 09/05/2024 02:02:33.529 7f59a971dd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.28:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.28:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.28:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/05/2024 02:02:33.550 7f59a971df60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.28:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.28:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.28:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 09/05/2024 02:02:33.579 7f59a974ce10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.28:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.28:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.28:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/05/2024 02:02:34.151 7f59b22ef148 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))",) 09/05/2024 02:02:34.655 7f59b22ef148 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 09/05/2024 02:02:34.656 7f59b22ef148 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 09/05/2024 02:02:34.790 7f59a970a978 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 09/05/2024 02:02:34.790 7f59a970a978 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 09/05/2024 02:02:34.791 7f59a8e3c438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.28:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.28:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.28:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.28,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 09/05/2024 02:02:35.028 7f59a8e8d4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.28:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.28:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.28:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.28,:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/05/2024 02:02:36.806 7f59a8d6f780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-85-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.28:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.28:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.28:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.28:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.28,:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/05/2024 02:03:46.084 7f59b22ef148 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 09/05/2024 02:03:46.084 7f59b22ef148 [contrail-device-manager] [WARNING]: initializing cassandra cluster 09/05/2024 02:03:46.085 7f59b22ef148 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/05/2024 02:03:46.087 7f59b22ef148 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/05/2024 02:03:46.119 7f59b22ef148 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/05/2024 02:03:46.136 7f59b22ef148 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 09/05/2024 02:03:46.148 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/05/2024 02:03:46.212 7f59b22ef148 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 09/05/2024 02:03:46.217 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/05/2024 02:03:46.223 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/05/2024 02:03:46.261 7f59b22ef148 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 09/05/2024 02:03:46.269 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/05/2024 02:03:46.269 7f59b22ef148 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 09/05/2024 02:03:46.305 7f59b22ef148 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 09/05/2024 02:03:46.312 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/05/2024 02:03:46.420 7f59b22ef148 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 09/05/2024 02:03:46.424 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/05/2024 02:03:46.430 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/05/2024 02:03:46.493 7f59b22ef148 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 09/05/2024 02:03:46.496 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/05/2024 02:03:46.500 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/05/2024 02:03:46.593 7f59b22ef148 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 09/05/2024 02:03:46.598 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/05/2024 02:03:46.603 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/05/2024 02:03:46.921 7f59b22ef148 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 09/05/2024 02:03:46.927 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/05/2024 02:03:46.939 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/05/2024 02:03:47.024 7f59b22ef148 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 09/05/2024 02:03:47.027 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/05/2024 02:03:47.033 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/05/2024 02:03:47.095 7f59b22ef148 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 09/05/2024 02:03:47.098 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/05/2024 02:03:47.105 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/05/2024 02:03:47.204 7f59b22ef148 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 09/05/2024 02:03:47.209 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/05/2024 02:03:47.220 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/05/2024 02:03:47.274 7f59b22ef148 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 09/05/2024 02:03:47.277 7f59b22ef148 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/05/2024 02:03:47.301 7f59b22ef148 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.28:5673//'] 09/05/2024 02:03:47.302 7f59b22ef148 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 09/05/2024 02:03:47.313 7f59b22ef148 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/05/2024 02:03:47.321 7f59b22ef148 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 09/05/2024 02:03:47.341 7f59b22ef148 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/05/2024 02:03:47.346 7f59a8d68eb8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 09/05/2024 02:03:47.367 7f59a8d68eb8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED