10/17/2024 02:11:28.834 7f56b6434248 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port
10/17/2024 02:11:28.834 7f56b6434248 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103
10/17/2024 02:11:28.837 7f56b6434448 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-101-1.cluster.local  process_status = [ <<  module_id = contrail-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
10/17/2024 02:11:28.838 7f56b6434448 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-101-1.cluster.local:Database:contrail-database-nodemgr:0  client_info = <<  status = Idle  successful_connections = 0  pid = 1  http_port = 8103  start_time = 1729131088837632  collector_name =   collector_ip =   collector_list = [ 10.0.0.33:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 1  >>  
10/17/2024 02:11:29.497 7f56b6434848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-101-1.cluster.local  build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2024-10-17 01:09:58.470181", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-6040-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}  system_cpu_info = <<  num_socket = 1  num_cpu = 4  num_core_per_socket = 4  num_thread_per_core = 1  >>  running_package_version = nightly-nightly.el7  installed_package_version = nightly-nightly.el7  >>  
10/17/2024 02:11:29.505 7f56b6434848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-101-1.cluster.local  process_status = [ <<  module_id = contrail-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
10/17/2024 02:11:29.506 7f56b6434848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-101-1.cluster.local  deleted = False  process_info = [ <<  process_name = cassandra  process_state = PROCESS_STATE_EXITED  start_count = 0  stop_count = 0  exit_count = 0  last_start_time =   last_stop_time =   last_exit_time =   core_file_list = [  ]  >>, <<  process_name = contrail-query-engine  process_state = PROCESS_STATE_EXITED  start_count = 0  stop_count = 0  exit_count = 0  last_start_time =   last_stop_time =   last_exit_time =   core_file_list = [  ]  >>, <<  process_name = contrail-database-nodemgr  process_state = PROCESS_STATE_RUNNING  start_count = 1  stop_count = 0  exit_count = 0  last_start_time = 1729131087000000  last_stop_time =   last_exit_time =   core_file_list = [  ]  >>,  ]  build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2024-10-17 01:09:58.470181", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-6040-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}  >>  
10/17/2024 02:11:30.459 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get database usage, will retry: cassandra
10/17/2024 02:11:31.467 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get database usage, will retry: cassandra
10/17/2024 02:11:32.271 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get database usage, will retry: cassandra
10/17/2024 02:11:32.893 7f56b6434848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-101-1.cluster.local  process_status = [ <<  module_id = contrail-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.33:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>,  ]  description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down  >>,  ]  >>  
10/17/2024 02:11:32.943 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get database usage, giving up: cassandra
10/17/2024 02:11:33.289 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: cassandra
10/17/2024 02:11:33.604 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: cassandra
10/17/2024 02:11:33.619 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:12:30.945 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1
10/17/2024 02:12:34.202 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:13:33.646 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:14:33.643 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:15:33.208 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:16:33.954 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:17:33.563 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:18:33.482 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:19:33.638 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:20:33.591 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:21:34.426 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:22:33.561 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:23:33.674 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:24:33.673 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:25:33.348 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:26:33.407 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:27:33.579 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:28:33.674 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:29:33.493 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:30:33.644 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:31:33.937 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:32:33.670 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:33:33.584 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:34:33.654 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:35:33.238 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:36:33.707 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:37:33.496 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:38:33.666 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:39:33.596 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
10/17/2024 02:40:33.536 7f56bc723178 [contrail-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'