09/19/2024 02:07:27.015 7fe8dc808b48 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port
09/19/2024 02:07:27.016 7fe8dc808b48 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103
09/19/2024 02:07:27.024 7fe8dc3b0448 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-89-1.cluster.local  process_status = [ <<  module_id = contrail-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
09/19/2024 02:07:27.024 7fe8dc3b0448 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-89-1.cluster.local:Database:contrail-database-nodemgr:0  client_info = <<  status = Idle  successful_connections = 0  pid = 1  http_port = 8103  start_time = 1726711647023681  collector_name =   collector_ip =   collector_list = [ 10.0.0.34:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 1  >>  
09/19/2024 02:07:27.997 7fe8dc3b0848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-89-1.cluster.local  build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2024-09-19 01:19:55.632943", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-5603-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}  system_cpu_info = <<  num_socket = 1  num_cpu = 4  num_core_per_socket = 4  num_thread_per_core = 1  >>  running_package_version = nightly-nightly.el7  installed_package_version = nightly-nightly.el7  >>  
09/19/2024 02:07:28.001 7fe8dc3b0848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-89-1.cluster.local  process_status = [ <<  module_id = contrail-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
09/19/2024 02:07:28.001 7fe8dc3b0848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-89-1.cluster.local  deleted = False  process_info = [ <<  process_name = cassandra  process_state = PROCESS_STATE_EXITED  start_count = 0  stop_count = 0  exit_count = 0  last_start_time =   last_stop_time =   last_exit_time =   core_file_list = [  ]  >>, <<  process_name = contrail-query-engine  process_state = PROCESS_STATE_EXITED  start_count = 0  stop_count = 0  exit_count = 0  last_start_time =   last_stop_time =   last_exit_time =   core_file_list = [  ]  >>, <<  process_name = contrail-database-nodemgr  process_state = PROCESS_STATE_RUNNING  start_count = 1  stop_count = 0  exit_count = 0  last_start_time = 1726711645000000  last_stop_time =   last_exit_time =   core_file_list = [  ]  >>,  ]  build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2024-09-19 01:19:55.632943", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-5603-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}  >>  
09/19/2024 02:07:29.355 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get database usage, will retry: cassandra
09/19/2024 02:07:29.952 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get database usage, will retry: cassandra
09/19/2024 02:07:30.240 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get database usage, will retry: cassandra
09/19/2024 02:07:30.688 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get database usage, giving up: cassandra
09/19/2024 02:07:31.024 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: cassandra
09/19/2024 02:07:31.030 7fe8dc3b0848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-89-1.cluster.local  process_status = [ <<  module_id = contrail-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.34:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>,  ]  description = Cassandra state detected DOWN. Disk space for DB not retrievable. Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
09/19/2024 02:07:31.030 7fe8dc3b0848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-89-1.cluster.local  process_status = [ <<  module_id = contrail-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.34:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>,  ]  description = Cassandra state detected DOWN. Disk space for DB not retrievable. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down  >>,  ]  >>  
09/19/2024 02:07:31.291 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: cassandra
09/19/2024 02:07:31.302 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:08:29.253 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1
09/19/2024 02:08:32.485 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:09:31.705 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:10:31.631 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:11:31.547 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:12:32.808 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:13:31.689 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:14:31.876 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:15:31.748 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:16:31.776 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:17:32.644 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:18:32.175 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:19:32.438 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:20:31.885 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:21:32.089 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:22:32.042 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:23:32.108 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:24:32.032 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:25:32.168 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:26:32.073 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:27:32.324 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:28:31.923 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:29:32.279 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:30:31.799 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:31:31.948 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:32:32.103 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:33:31.938 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:34:31.869 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:35:31.828 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:36:31.758 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:37:32.247 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:38:31.892 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:39:31.888 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:40:31.844 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:41:32.090 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:42:31.894 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:43:32.160 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:44:32.346 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
09/19/2024 02:45:32.203 7fe8e207e898 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'