12/01/2025 01:34:37.524 7f79cfa90400 [contrail-config-database-nodemgr] [ERROR]: Cannot write http_port 8112 to /tmp/contrail-config-database-nodemgr.0.http_port
12/01/2025 01:34:37.524 7f79cfa90400 [contrail-config-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8112
12/01/2025 01:34:37.526 7f79cfa907c0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-275-1.cluster.local  process_status = [ <<  module_id = contrail-config-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
12/01/2025 01:34:37.526 7f79cfa907c0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-275-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0  client_info = <<  status = Idle  successful_connections = 0  pid = 1  http_port = 8112  start_time = 1764552877525953  collector_name =   collector_ip =   collector_list = [ 10.0.0.249:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 1  >>  
12/01/2025 01:34:37.764 7f79cfa90cc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-275-1.cluster.local  build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2025-12-01 00:48:28.850228", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-12899-1.novalocal", "build-id": "v0+nightly", "build-number": "202512010007"}]}  system_cpu_info = <<  num_socket = 1  num_cpu = 4  num_core_per_socket = 4  num_thread_per_core = 1  >>  >>  
12/01/2025 01:34:37.768 7f79cfa90cc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-275-1.cluster.local  process_status = [ <<  module_id = contrail-config-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
12/01/2025 01:34:37.768 7f79cfa90cc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-275-1.cluster.local  deleted = False  process_info = [ <<  process_name = cassandra  process_state = PROCESS_STATE_RUNNING  start_count = 1  stop_count = 0  exit_count = 0  last_start_time = 1764552842000000  last_stop_time =   last_exit_time =   core_file_list = [  ]  >>, <<  process_name = zookeeper  process_state = PROCESS_STATE_RUNNING  start_count = 1  stop_count = 0  exit_count = 0  last_start_time = 1764552832000000  last_stop_time =   last_exit_time =   core_file_list = [  ]  >>, <<  process_name = contrail-config-database-nodemgr  process_state = PROCESS_STATE_RUNNING  start_count = 1  stop_count = 0  exit_count = 0  last_start_time = 1764552876000000  last_stop_time =   last_exit_time =   core_file_list = [  ]  >>,  ]  build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2025-12-01 00:48:28.850228", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-12899-1.novalocal", "build-id": "v0+nightly", "build-number": "202512010007"}]}  >>  
12/01/2025 01:34:38.456 7f79cfa90e00 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-275-1.cluster.local  database_usage = [ <<  disk_space_used_1k = 12080824  disk_space_available_1k = 70610228  config_db_size_1k = 472  >>,  ]  >>  
12/01/2025 01:34:38.906 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1
12/01/2025 01:34:38.908 7f79cfa90cc0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-275-1.cluster.local  process_status = [ <<  module_id = contrail-config-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>,  ]  description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down  >>,  ]  >>  
12/01/2025 01:34:41.554 7f79cfa90860 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-275-1.cluster.local  process_status = [ <<  module_id = contrail-config-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.249:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>,  ]  description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down  >>,  ]  >>  
12/01/2025 01:34:43.455 7f79cf992540 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed
12/01/2025 01:34:43.457 7f79cfca14a0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-275-1.cluster.local  process_status = [ <<  module_id = contrail-config-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.249:8086,  ]  status = Down  description = ClientInit to Idle on EvTcpClose  >>,  ]  description = Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down  >>,  ]  >>  
12/01/2025 01:34:43.458 7f79cfca14a0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-275-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0  client_info = <<  status = Idle  successful_connections = 1  pid = 1  http_port = 8112  start_time = 1764552877525953  collector_name =   collector_ip = 10.0.0.249:8086  collector_list = [ 10.0.0.249:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 2  >>  
12/01/2025 01:34:45.569 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr config_db_uuid'. Error is: Result: 1
12/01/2025 01:34:47.468 7f79d07d7540 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = cn-jenkins-deploy-platform-k8s-manifests-275-1.cluster.local  process_status = [ <<  module_id = contrail-config-database-nodemgr  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name = Collector  server_addrs = [ 10.0.0.249:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>,  ]  description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down  >>,  ]  >>  
12/01/2025 01:34:47.738 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr to_bgp_keyspace'. Error is: Result: 1
12/01/2025 01:34:49.586 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr svc_monitor_keyspace'. Error is: Result: 1
12/01/2025 01:34:51.612 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr DISCOVERY_SERVER'. Error is: Result: 1
12/01/2025 01:34:55.973 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr useragent'. Error is: Result: 1
12/01/2025 01:34:58.521 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr dm_keyspace'. Error is: Result: 1
12/01/2025 01:34:58.539 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/01/2025 01:35:44.781 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/01/2025 01:36:41.804 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/01/2025 01:37:42.960 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/01/2025 01:38:41.614 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/01/2025 01:39:41.691 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/01/2025 01:40:41.710 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/01/2025 01:41:41.947 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/01/2025 01:42:43.467 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/01/2025 01:43:41.578 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'
12/01/2025 01:44:42.748 7f79d29ac140 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'