01/16/2025 01:57:29.568 7fdbe2392048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-127-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/16/2025 01:57:29.569 7fdbe2392048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-127-1.cluster.local:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 111 start_time = 1736992649568728 collector_name = collector_ip = collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 01/16/2025 01:57:29.618 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.12:5673//'] 01/16/2025 01:57:29.628 7fdbeaeb7688 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 01/16/2025 01:57:29.629 7fdbe2263d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-127-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 01/16/2025 01:57:29.629 7fdbe2263d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-127-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 01/16/2025 01:57:29.647 7fdbeaeb7688 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 01/16/2025 01:57:29.648 7fdbe2268b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-127-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/16/2025 01:57:29.685 7fdbeaeb7688 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 01/16/2025 01:57:29.686 7fdbe180af60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-127-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 01/16/2025 01:57:29.713 7fdbeaeb7688 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 01/16/2025 01:57:29.718 7fdbe180a780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-127-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/16/2025 01:57:29.749 7fdbeaeb7688 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 01/16/2025 01:57:29.755 7fdbeaeb7688 [contrail-device-manager] [WARNING]: initializing cassandra cluster 01/16/2025 01:57:29.755 7fdbeaeb7688 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 01/16/2025 01:57:29.763 7fdbe180a0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-127-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.12:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 01/16/2025 01:57:29.774 7fdbeaeb7688 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 01/16/2025 01:57:30.281 7fdbeaeb7688 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 01/16/2025 01:57:30.306 7fdbe22df240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-127-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.12:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 01/16/2025 01:57:30.874 7fdbeaeb7688 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 01/16/2025 01:57:30.874 7fdbe22cd048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-127-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.12:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 01/16/2025 01:57:31.934 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 01/16/2025 01:57:33.180 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 01/16/2025 01:57:33.579 7fdbe22cf4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-127-1.cluster.local process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.12:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.12:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 01/16/2025 01:57:34.284 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 01/16/2025 01:57:35.302 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 01/16/2025 01:57:36.562 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 01/16/2025 01:57:37.642 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 01/16/2025 01:57:38.677 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 01/16/2025 01:57:39.752 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 01/16/2025 01:57:40.766 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 01/16/2025 01:57:41.779 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 01/16/2025 01:57:41.892 7fdbeaeb7688 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 01/16/2025 01:57:42.068 7fdbeaeb7688 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 01/16/2025 01:57:42.514 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:57:42.624 7fdbeaeb7688 [contrail-device-manager] [WARNING]: table 'obj_uuid_table', created 01/16/2025 01:57:42.632 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:57:43.287 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:57:43.648 7fdbeaeb7688 [contrail-device-manager] [WARNING]: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000} 01/16/2025 01:57:43.668 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:57:44.299 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:57:44.304 7fdbeaeb7688 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 01/16/2025 01:57:44.311 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:57:45.570 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:57:45.688 7fdbeaeb7688 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 01/16/2025 01:57:45.698 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:57:46.500 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:57:46.511 7fdbeaeb7688 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 01/16/2025 01:57:46.521 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:57:47.293 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:57:47.681 7fdbeaeb7688 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 01/16/2025 01:57:47.685 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:57:48.253 7fdbeaeb7688 [contrail-device-manager] [ERROR]: ZTP manager: Error restarting dnsmasq DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))",) 01/16/2025 01:57:48.755 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Process 111 prepared to run in Full mode... 01/16/2025 01:57:48.755 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 01/16/2025 01:57:49.160 7fdbe2131978 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 01/16/2025 01:57:49.160 7fdbe2131978 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 01/16/2025 01:57:55.549 7fdbeaeb7688 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 01/16/2025 01:57:55.549 7fdbeaeb7688 [contrail-device-manager] [ERROR]: Unable to load init data 01/16/2025 01:57:55.549 7fdbeaeb7688 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 01/16/2025 01:57:55.550 7fdbeaeb7688 [contrail-device-manager] [WARNING]: initializing cassandra cluster 01/16/2025 01:57:55.550 7fdbeaeb7688 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 01/16/2025 01:57:55.552 7fdbeaeb7688 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 01/16/2025 01:57:55.720 7fdbeaeb7688 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 01/16/2025 01:57:55.744 7fdbeaeb7688 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 01/16/2025 01:57:55.796 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:57:55.987 7fdbeaeb7688 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 01/16/2025 01:57:55.990 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:57:56.000 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:57:56.075 7fdbeaeb7688 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 01/16/2025 01:57:56.078 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:57:56.078 7fdbeaeb7688 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 01/16/2025 01:57:56.115 7fdbeaeb7688 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 01/16/2025 01:57:56.254 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:57:56.483 7fdbeaeb7688 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 01/16/2025 01:57:56.493 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:57:57.089 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:57:57.303 7fdbeaeb7688 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 01/16/2025 01:57:57.326 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:57:57.607 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:57:57.807 7fdbeaeb7688 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 01/16/2025 01:57:57.814 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:57:58.082 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:57:58.204 7fdbeaeb7688 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 01/16/2025 01:57:58.209 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:57:58.629 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:57:58.915 7fdbeaeb7688 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 01/16/2025 01:57:58.931 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:57:59.454 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:57:59.673 7fdbeaeb7688 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 01/16/2025 01:57:59.684 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:58:00.306 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:58:00.585 7fdbeaeb7688 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 01/16/2025 01:58:00.593 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:58:01.406 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 01/16/2025 01:58:01.591 7fdbeaeb7688 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 01/16/2025 01:58:01.602 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Zookeeper lock released. 01/16/2025 01:58:01.807 7fdbeaeb7688 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.12:5673//'] 01/16/2025 01:58:01.807 7fdbeaeb7688 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 01/16/2025 01:58:01.837 7fdbeaeb7688 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 01/16/2025 01:58:01.853 7fdbeaeb7688 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 01/16/2025 01:58:01.874 7fdbeaeb7688 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 01/16/2025 01:58:01.883 7fdbe1bfe048 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 01/16/2025 01:58:01.915 7fdbe1bfe048 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 01/16/2025 01:58:02.356 7fdbe1c02eb8 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 01/16/2025 01:58:02.356 7fdbe1c02eb8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 01/16/2025 01:58:02.376 7fdbe1c02eb8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 01/16/2025 01:58:02.382 7fdbe1c02eb8 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 01/16/2025 01:58:02.382 7fdbe1c02eb8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 01/16/2025 01:58:02.390 7fdbe1c02eb8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 01/16/2025 01:58:02.397 7fdbe1c02eb8 [contrail-device-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'DeviceManagerLogger' object has no attribute 'warn' 01/16/2025 01:58:02.426 7fdbe1c02eb8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 01/16/2025 01:58:02.470 7fdbe1c02eb8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED