07/10/2025 01:44:33.834 7fe5591098d0 [contrail-config-database-nodemgr] [ERROR]: Cannot write http_port 8112 to /tmp/contrail-config-database-nodemgr.0.http_port 07/10/2025 01:44:33.835 7fe5591098d0 [contrail-config-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8112 07/10/2025 01:44:33.836 7fe559110240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/10/2025 01:44:33.837 7fe559110240 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1752111873836512 collector_name = collector_ip = collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 07/10/2025 01:44:34.423 7fe5619d6a20 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-07-10 00:46:04.071018", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9876-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 07/10/2025 01:44:34.426 7fe55903eb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/10/2025 01:44:34.426 7fe55903eb70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1752111871000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_EXITED start_count = 0 stop_count = 0 exit_count = 0 last_start_time = last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1752111872000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-07-10 00:46:04.071018", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9876-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 07/10/2025 01:44:35.338 7fe55ccef4e0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local database_usage = [ << disk_space_used_1k = 8777784 disk_space_available_1k = 73913268 config_db_size_1k = 4 >>, ] >> 07/10/2025 01:44:35.997 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1 07/10/2025 01:44:36.002 7fe559048ac8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/10/2025 01:44:37.838 7fe559048198 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/10/2025 01:44:37.841 7fe559048198 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/10/2025 01:44:37.842 7fe559048198 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/10/2025 01:44:37.842 7fe559048198 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1752111873836512 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/10/2025 01:44:40.818 7fe55ccef2e8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] CassandraStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local cassandra_compaction_task = << pending_compaction_tasks = 0 >> thread_pool_stats = [ << pool_name = MemtableFlushWriter active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = CompactionExecutor active = 0 pending = 0 all_time_blocked = 0 >>, ] >> 07/10/2025 01:44:41.840 7fe553b3d390 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/10/2025 01:44:41.841 7fe553b3d390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/10/2025 01:44:41.841 7fe553b3d390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/10/2025 01:44:41.841 7fe553b3d390 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1752111873836512 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/10/2025 01:44:42.748 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr config_db_uuid'. Error is: Result: 1 07/10/2025 01:44:44.257 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr to_bgp_keyspace'. Error is: Result: 1 07/10/2025 01:44:45.356 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr svc_monitor_keyspace'. Error is: Result: 1 07/10/2025 01:44:45.844 7fe55910ef60 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/10/2025 01:44:45.845 7fe55910ef60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/10/2025 01:44:45.847 7fe55910ef60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/10/2025 01:44:45.847 7fe55910ef60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1752111873836512 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/10/2025 01:44:46.483 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr DISCOVERY_SERVER'. Error is: Result: 1 07/10/2025 01:44:47.967 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr useragent'. Error is: Result: 1 07/10/2025 01:44:49.365 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7201 repair -pr dm_keyspace'. Error is: Result: 1 07/10/2025 01:44:49.373 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:44:49.851 7fe55902e0f0 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/10/2025 01:44:49.851 7fe55902e0f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/10/2025 01:44:49.852 7fe55902e0f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/10/2025 01:44:49.852 7fe55902e0f0 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1752111873836512 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/10/2025 01:44:49.963 7fe5614bff60 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1752111871000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = zookeeper process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1752111889960181 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-config-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1752111872000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-07-10 00:46:04.071018", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9876-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 07/10/2025 01:44:53.853 7fe55902eeb8 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/10/2025 01:44:53.856 7fe55902eeb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/10/2025 01:44:53.858 7fe55902eeb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/10/2025 01:44:53.859 7fe55902eeb8 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1752111873836512 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/10/2025 01:44:54.452 7fe55cd4fd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local disk_usage_info = { /dev/vda4 : << partition_type = xfs partition_space_used_1k = 9541748 partition_space_available_1k = 73149304 percentage_partition_space_used = 12 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1345620 cpu_share = 0.0 mem_res = 1272168 >>zookeeper : << mem_virt = 67572 cpu_share = 0.0 mem_res = 65640 >>contrail-config-database-nodemgr : << mem_virt = 50380 cpu_share = 0.0 mem_res = 40324 >> } system_mem_usage = << total = 16111716 used = 5465632 free = 2103772 buffers = 5136 cached = 8537176 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 4.5 five_min_avg = 1.43 fifteen_min_avg = 0.56 cpu_share = 0 node_type = config-database-node >> >> 07/10/2025 01:44:57.857 7fe55cd4fd68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/10/2025 01:44:57.858 7fe55cd4fd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/10/2025 01:44:57.858 7fe55cd4fd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/10/2025 01:44:57.858 7fe55cd4fd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1752111873836512 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/10/2025 01:45:01.863 7fe55cd4fd68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/10/2025 01:45:01.863 7fe55cd4fd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/10/2025 01:45:01.864 7fe55cd4fd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/10/2025 01:45:01.864 7fe55cd4fd68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1752111873836512 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/10/2025 01:45:05.865 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/10/2025 01:45:05.866 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/10/2025 01:45:05.866 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/10/2025 01:45:05.866 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1752111873836512 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/10/2025 01:45:09.872 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/10/2025 01:45:09.872 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/10/2025 01:45:09.873 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/10/2025 01:45:09.873 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1752111873836512 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/10/2025 01:45:13.877 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/10/2025 01:45:13.877 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/10/2025 01:45:13.878 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/10/2025 01:45:13.878 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1752111873836512 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/10/2025 01:45:17.882 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 07/10/2025 01:45:17.883 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/10/2025 01:45:17.883 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/10/2025 01:45:17.883 7fe55c8e4d68 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1752111873836512 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/10/2025 01:45:21.886 7fe55981db70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/10/2025 01:45:23.670 7fe55901ad68 [contrail-config-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 07/10/2025 01:45:23.671 7fe55981db70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 07/10/2025 01:45:23.671 7fe55981db70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8112 start_time = 1752111873836512 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/10/2025 01:45:27.684 7fe55981db70 [contrail-config-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-226-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/10/2025 01:45:49.129 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:46:42.226 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:47:40.541 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:48:39.819 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:49:41.027 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:50:38.987 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:51:39.186 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:52:38.989 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:53:39.063 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:54:38.962 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:55:39.856 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:56:39.727 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:57:39.252 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:58:39.411 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 01:59:39.505 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:00:39.289 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:01:39.412 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:02:39.357 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:03:40.257 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:04:39.998 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:05:39.793 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:06:39.693 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:07:40.159 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:08:39.584 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:09:39.561 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:10:39.943 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:11:39.800 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:12:39.710 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:13:39.516 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:14:39.826 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:15:39.794 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:16:39.716 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:17:39.765 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:18:39.643 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:19:39.086 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:20:40.048 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:21:39.709 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:22:40.268 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:23:40.102 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:24:39.824 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' 07/10/2025 02:25:40.162 7fe55b7a3988 [contrail-config-database-nodemgr] [ERROR]: Failed to get core files: [Errno 2] No such file or directory: '/var/crashes'