11/27/2025 01:35:39.582 7fa2aebe7400 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port
11/27/2025 01:35:39.583 7fa2aebe7400 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103
11/27/2025 01:35:39.586 7fa2aebe77c0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-274-1.cluster.local  process_status = [ <<  module_id = contrail-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
11/27/2025 01:35:39.587 7fa2aebe77c0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-274-1.cluster.local:Database:contrail-database-nodemgr:0  client_info = <<  status = Idle  successful_connections = 0  pid = 1  http_port = 8103  start_time = 1764207339585966  collector_name =   collector_ip =   collector_list = [ 10.0.0.249:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 1  >>  
11/27/2025 01:35:40.515 7fa2aebe7e00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-274-1.cluster.local  build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2025-11-27 00:50:11.016574", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-12893-1.novalocal", "build-id": "v0+nightly", "build-number": "202511270009"}]}  system_cpu_info = <<  num_socket = 1  num_cpu = 4  num_core_per_socket = 4  num_thread_per_core = 1  >>  >>  
11/27/2025 01:35:40.521 7fa2aebe7e00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-274-1.cluster.local  process_status = [ <<  module_id = contrail-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
11/27/2025 01:35:40.521 7fa2aebe7e00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-274-1.cluster.local  deleted = False  process_info = [ <<  process_name = cassandra  process_state = PROCESS_STATE_EXITED  start_count = 0  stop_count = 0  exit_count = 0  last_start_time =   last_stop_time =   last_exit_time =   core_file_list = [  ]  >>, <<  process_name = contrail-query-engine  process_state = PROCESS_STATE_EXITED  start_count = 0  stop_count = 0  exit_count = 0  last_start_time =   last_stop_time =   last_exit_time =   core_file_list = [  ]  >>, <<  process_name = contrail-database-nodemgr  process_state = PROCESS_STATE_RUNNING  start_count = 1  stop_count = 0  exit_count = 0  last_start_time = 1764207337000000  last_stop_time =   last_exit_time =   core_file_list = [  ]  >>,  ]  build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2025-11-27 00:50:11.016574", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-12893-1.novalocal", "build-id": "v0+nightly", "build-number": "202511270009"}]}  >>  
11/27/2025 01:35:40.979 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get database usage, will retry: cassandra
11/27/2025 01:35:41.142 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get database usage, will retry: cassandra
11/27/2025 01:35:41.308 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get database usage, will retry: cassandra
11/27/2025 01:35:41.502 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get database usage, giving up: cassandra
11/27/2025 01:35:41.635 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: cassandra
11/27/2025 01:35:41.637 7fa2aedf54a0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-274-1.cluster.local  process_status = [ <<  module_id = contrail-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Cassandra state detected DOWN. Disk space for DB not retrievable. Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
11/27/2025 01:35:41.786 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: cassandra
11/27/2025 01:35:41.800 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
11/27/2025 01:35:42.831 7fa2aedf5c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-274-1.cluster.local  disk_usage_info = { /dev/vda4 : <<  partition_type = xfs  partition_space_used_1k = 12084048  partition_space_available_1k = 70607004  percentage_partition_space_used = 15  >> }  process_mem_cpu_usage = { contrail-database-nodemgr : <<  mem_virt = 48888  cpu_share = 0.0  mem_res = 41380  >> }  system_mem_usage = <<  total = 16111712  used = 6055628  free = 294540  buffers = 4544  cached = 10129720  node_type = database-node  >>  system_cpu_usage = <<  one_min_avg = 4.74  five_min_avg = 1.72  fifteen_min_avg = 0.64  cpu_share = 0  node_type = database-node  >>  >>  
11/27/2025 01:35:43.587 7fa2aedf5c20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail
11/27/2025 01:35:43.588 7fa2aedf5c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-274-1.cluster.local  process_status = [ <<  module_id = contrail-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.249:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>,  ]  description = Cassandra state detected DOWN. Disk space for DB not retrievable. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down  >>,  ]  >>  
11/27/2025 01:35:43.588 7fa2aedf5c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-274-1.cluster.local  process_status = [ <<  module_id = contrail-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.249:8086,  ]  status = Down  description = Connect to Idle on EvTcpConnectFail  >>,  ]  description = Cassandra state detected DOWN. Disk space for DB not retrievable. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down  >>,  ]  >>  
11/27/2025 01:35:43.589 7fa2aedf5c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-274-1.cluster.local:Database:contrail-database-nodemgr:0  client_info = <<  status = Idle  successful_connections = 0  pid = 1  http_port = 8103  start_time = 1764207339585966  collector_name =   collector_ip = 10.0.0.249:8086  collector_list = [ 10.0.0.249:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 2  >>  
11/27/2025 01:35:47.595 7fa2aedf5c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-274-1.cluster.local  process_status = [ <<  module_id = contrail-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.249:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>,  ]  description = Cassandra state detected DOWN. Disk space for DB not retrievable. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down  >>,  ]  >>  
11/27/2025 01:36:42.069 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1
11/27/2025 01:36:45.271 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
11/27/2025 01:37:44.264 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
11/27/2025 01:38:44.908 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
11/27/2025 01:39:44.245 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
11/27/2025 01:40:46.306 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
11/27/2025 01:41:44.486 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
11/27/2025 01:42:44.368 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
11/27/2025 01:43:44.446 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
11/27/2025 01:44:46.090 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
11/27/2025 01:45:45.154 7fa2b1b00280 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'