01/13/2025 02:00:17.845 7ff5ec0cf438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/13/2025 02:00:17.846 7ff5ec0cf438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 111 start_time = 1736733617845263 collector_name = collector_ip = collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 01/13/2025 02:00:18.048 7ff5f49fc508 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.27:5673//'] 01/13/2025 02:00:18.125 7ff5f49fc508 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 01/13/2025 02:00:18.126 7ff5e7ba1cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.27:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 01/13/2025 02:00:18.126 7ff5e7ba1cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.27:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 01/13/2025 02:00:18.144 7ff5f49fc508 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 01/13/2025 02:00:18.146 7ff5e7333588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.27:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/13/2025 02:00:18.457 7ff5f49fc508 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 01/13/2025 02:00:18.460 7ff5e7337978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.27:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 01/13/2025 02:00:18.513 7ff5f49fc508 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 01/13/2025 02:00:18.514 7ff5e7337978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.27:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/13/2025 02:00:18.582 7ff5f49fc508 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 01/13/2025 02:00:18.583 7ff5f49fc508 [contrail-device-manager] [WARNING]: initializing cassandra cluster 01/13/2025 02:00:18.583 7ff5f49fc508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 01/13/2025 02:00:18.594 7ff5e7337048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.27:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.27:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 01/13/2025 02:00:18.596 7ff5f49fc508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 01/13/2025 02:00:18.730 7ff5f49fc508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 01/13/2025 02:00:18.737 7ff5e7d44b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.27:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.27:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.27:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 01/13/2025 02:00:18.787 7ff5f49fc508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 01/13/2025 02:00:18.787 7ff5ec05e978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.27:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.27:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.27:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 01/13/2025 02:00:18.799 7ff5f49fc508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 01/13/2025 02:00:18.834 7ff5f49fc508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 01/13/2025 02:00:20.143 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/13/2025 02:00:20.159 7ff5f49fc508 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 01/13/2025 02:00:20.167 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/13/2025 02:00:20.452 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/13/2025 02:00:20.682 7ff5f49fc508 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 01/13/2025 02:00:20.691 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/13/2025 02:00:20.693 7ff5e7d49eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.27:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.27:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.27:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/13/2025 02:00:20.711 7ff5e7d60978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.27:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.27:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.27:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 01/13/2025 02:00:20.728 7ff5e7d60978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.27:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.27:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.27:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/13/2025 02:00:21.238 7ff5f49fc508 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))",) 01/13/2025 02:00:21.740 7ff5f49fc508 [contrail-device-manager] [WARNING]: Process 111 prepared to run in Full mode... 01/13/2025 02:00:21.740 7ff5f49fc508 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 01/13/2025 02:00:22.310 7ff5e7d5f588 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 01/13/2025 02:00:22.310 7ff5e7d5f588 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 01/13/2025 02:00:22.310 7ff5e780cb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.27:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.27:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.27:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.27,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 01/13/2025 02:00:22.315 7ff5e780cb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-126-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.27:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.27:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.27:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.27,:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 01/13/2025 02:00:28.441 7ff5f49fc508 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 01/13/2025 02:00:28.441 7ff5f49fc508 [contrail-device-manager] [ERROR]: Unable to load init data 01/13/2025 02:00:28.441 7ff5f49fc508 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 01/13/2025 02:00:28.441 7ff5f49fc508 [contrail-device-manager] [WARNING]: initializing cassandra cluster 01/13/2025 02:00:28.441 7ff5f49fc508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 01/13/2025 02:00:28.452 7ff5f49fc508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 01/13/2025 02:00:28.613 7ff5f49fc508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 01/13/2025 02:00:28.637 7ff5f49fc508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 01/13/2025 02:00:28.819 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/13/2025 02:00:28.917 7ff5f49fc508 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 01/13/2025 02:00:28.928 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/13/2025 02:00:29.042 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/13/2025 02:00:29.179 7ff5f49fc508 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 01/13/2025 02:00:29.190 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/13/2025 02:00:29.190 7ff5f49fc508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 01/13/2025 02:00:29.348 7ff5f49fc508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 01/13/2025 02:00:29.466 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/13/2025 02:00:29.638 7ff5f49fc508 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 01/13/2025 02:00:29.643 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/13/2025 02:00:30.035 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/13/2025 02:00:30.194 7ff5f49fc508 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 01/13/2025 02:00:30.216 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/13/2025 02:00:30.569 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/13/2025 02:00:30.824 7ff5f49fc508 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 01/13/2025 02:00:30.835 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/13/2025 02:00:31.219 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/13/2025 02:00:31.449 7ff5f49fc508 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 01/13/2025 02:00:31.460 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/13/2025 02:00:31.746 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/13/2025 02:00:32.017 7ff5f49fc508 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 01/13/2025 02:00:32.028 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/13/2025 02:00:32.212 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/13/2025 02:00:32.988 7ff5f49fc508 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 01/13/2025 02:00:32.999 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/13/2025 02:00:33.323 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/13/2025 02:00:33.613 7ff5f49fc508 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 01/13/2025 02:00:33.620 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/13/2025 02:00:34.105 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/13/2025 02:00:34.314 7ff5f49fc508 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 01/13/2025 02:00:34.322 7ff5f49fc508 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/13/2025 02:00:34.399 7ff5f49fc508 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.27:5673//'] 01/13/2025 02:00:34.404 7ff5f49fc508 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 01/13/2025 02:00:34.474 7ff5f49fc508 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 01/13/2025 02:00:34.662 7ff5f49fc508 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 01/13/2025 02:00:34.761 7ff5f49fc508 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 01/13/2025 02:00:34.785 7ff5e7732048 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 01/13/2025 02:00:34.907 7ff5e7732048 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED