INFO: =================== Mon Sep 8 01:42:41 AM UTC 2025 =================== /contrail-functions.sh: line 167: /etc/contrail/vnc_api_lib.ini.tmp: No such file or directory /contrail-functions.sh: line 202: /etc/contrail/vnc_api_lib.ini.tmp: No such file or directory mv: cannot stat '/etc/contrail/vnc_api_lib.ini.tmp': No such file or directory [DEFAULTS] http_server_ip=0.0.0.0 log_file=/var/log/contrail/config-database-nodemgr/contrail-config-database-nodemgr.log log_level=SYS_NOTICE log_local=1 hostip=10.0.0.50 db_port=9041 db_jmx_port=7201 db_use_ssl=False [COLLECTOR] server_list=10.0.0.50:8086 [SANDESH] introspect_ssl_enable=False sandesh_ssl_enable=False [DEFAULTS] minimum_diskGB = 2 09/08/2025 01:42:42.465 7f3ba95a7540 [contrail-config-database-nodemgr] [INFO]: SANDESH: CONNECT TO COLLECTOR: True 09/08/2025 01:42:42.695 7f3ba95a7540 [contrail-config-database-nodemgr] [ERROR]: Failed to import package "database.sandesh" 09/08/2025 01:42:42.705 7f3ba95a7540 [contrail-config-database-nodemgr] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8112 09/08/2025 01:42:42.709 7f3ba95a7540 [contrail-config-database-nodemgr] [ERROR]: Failed to import package "database.sandesh" 09/08/2025 01:42:42.716 7f3ba95a7540 [contrail-config-database-nodemgr] [INFO]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_NOTICE] CRITICAL:root:Exit data: {'ID': 'f86794810337b26a669fd92baa19e954449f0a83ca3b41b325d26a710174f3d8', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'df', 'arguments': ['/var/lib/cassandra/data'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 69217} ERROR:contrail-config-database-nodemgr:Failed to get database usage, will retry: Result: 1 ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local database_usage = [ << disk_space_used_1k = 9635808 disk_space_available_1k = 73055244 config_db_size_1k = 0 >>, ] >> CRITICAL:root:Exit data: {'ID': '881ab688e6b13d304b013417e52dbe341d2a0e796a39f2a6099260ec598ccbe2', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.50', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 69290} ERROR:contrail-config-database-nodemgr:Failed to connect to database by CQL and socket: Result: 1 ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> CRITICAL:root:Exit data: {'ID': '4f3a3ca0683ddd2efbab3c67da583c041cc4d1cdcd8f1385e392a07a90cd0661', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 69320} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 CRITICAL:root:Exit data: {'ID': '4839cc29d1dfa8f52247492a5a2b5a5d987276aa9f38a034f1f90d129551fdec', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'repair', '-pr', 'config_db_uuid'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 69525} ERROR:contrail-config-database-nodemgr:Failed to run cmd: 'nodetool -p 7201 repair -pr config_db_uuid'. Error is: Result: 1 ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757295762731255 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> CRITICAL:root:Exit data: {'ID': '70b049ca3458536b8e57ba941fc001ae9bd905cc4fe6aca5b1598fc73ed0b4cf', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'repair', '-pr', 'to_bgp_keyspace'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 69619} ERROR:contrail-config-database-nodemgr:Failed to run cmd: 'nodetool -p 7201 repair -pr to_bgp_keyspace'. Error is: Result: 1 CRITICAL:root:Exit data: {'ID': 'dfbca1c2a22b3450961f11fb228021ec21bd6241a00043d793b436f05693dc1f', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'repair', '-pr', 'svc_monitor_keyspace'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 69705} ERROR:contrail-config-database-nodemgr:Failed to run cmd: 'nodetool -p 7201 repair -pr svc_monitor_keyspace'. Error is: Result: 1 CRITICAL:root:Exit data: {'ID': 'f2dd8aea68498dd021997b9cfad83a9b9618fbc0c9d23248e249f583b0838c5d', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'repair', '-pr', 'DISCOVERY_SERVER'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 69778} ERROR:contrail-config-database-nodemgr:Failed to run cmd: 'nodetool -p 7201 repair -pr DISCOVERY_SERVER'. Error is: Result: 1 CRITICAL:root:Exit data: {'ID': '1f37fd56e8ce40e7afb50eb22bb5b7daa167b177b2403d0bce0037583a3ff6df', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'repair', '-pr', 'useragent'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 69853} ERROR:contrail-config-database-nodemgr:Failed to run cmd: 'nodetool -p 7201 repair -pr useragent'. Error is: Result: 1 CRITICAL:root:Exit data: {'ID': '039851e3fe3e7899895f398c89d86edfc79d67837ddb1337e1239c69186cba6b', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'repair', '-pr', 'dm_keyspace'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 70019} ERROR:contrail-config-database-nodemgr:Failed to run cmd: 'nodetool -p 7201 repair -pr dm_keyspace'. Error is: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757295762731255 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757295762731255 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local disk_usage_info = { /dev/vda4 : << partition_type = xfs partition_space_used_1k = 9856764 partition_space_available_1k = 72834288 percentage_partition_space_used = 12 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1232612 cpu_share = 0.0 mem_res = 1223580 >>zookeeper : << mem_virt = 59956 cpu_share = 0.0 mem_res = 57656 >>contrail-config-database-nodemgr : << mem_virt = 53144 cpu_share = 0.0 mem_res = 44900 >> } system_mem_usage = << total = 16111716 used = 5556932 free = 1316940 buffers = 5136 cached = 9232708 node_type = config-database-node >> system_cpu_usage = << one_min_avg = 3.45 five_min_avg = 1.1 fifteen_min_avg = 0.41 cpu_share = 0 node_type = config-database-node >> >> ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757295762731255 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757295762731255 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757295762731255 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757295762731255 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757295762731255 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:Session Event: TCP Connect Fail ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local:ConfigDatabase:contrail-config-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8112 start_time = 1757295762731255 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> ERROR:contrail-config-database-nodemgr:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-247-1.cluster.local process_status = [ << module_id = contrail-config-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> CRITICAL:root:Exit data: {'ID': '84c80ecc2e4b422153f6811d644aa9e03280aacacd561016b077f19ed6137feb', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.50', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 76974} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': '8030c573f9b72cb5eeb5fdb456945336e614fe4af14a9e501f5fef54288a9cdc', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 77032} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' [Introspect:8112]10.0.0.50 - - [2025-09-08 01:44:40] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 6735 0.001240 [Introspect:8112]10.0.0.50 - - [2025-09-08 01:44:42] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 6735 0.000977 CRITICAL:root:Exit data: {'ID': '5f99a250c60606f3aabdc9047be23f337cc7e9048c10af7d1731d4342871b585', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.50', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 79350} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': '49aec10204c5093b139b0ef7831f35b55845f87611e3c436c40a2fff7f2bde6b', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 79368} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': 'a5a646a8d416bcc114e03aadbf35627f457cc8e065918af0665326f5c8063052', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.50', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 81032} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': 'dc4099cb074b792b31dda753fcaa2049296d29b452b524c53709005c2b7d6d38', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 81050} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': 'c495813eeb52eed2d31a37cbd7ee73bbcf53be349e5a42e2f60c8bfed109c837', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.50', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 83371} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': '7e9c9e44f04b891440e989f42d0412541cac41e0c4522b568e493348d24d0a38', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 83390} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '749dd6bb292042fcd294a33844bf53b3ffd460189b6517f9d8256e39574e2611', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.50', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 86494} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': '0ed3a776079f304eb7e841f734f78a1f795f3fdcb5434901bd04bce030ff7ef8', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 86513} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '7c5ee87ed690c6c584cf46793eeb18a06d0cdd4295ec7a899ba72ce10024fca7', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.50', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 87686} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': '9d930c1e3e1f8198522b41e104592a62879f3488242f2dac328269653382ce06', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 87704} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '44473f6e28ab3f7c1d5bcde43ad31963bd218ddb9316dccea67674203202a2f7', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.50', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 88777} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': 'a380f887e145db39f091fcea7771f566c347d4af292bd8b471546b289faeeabc', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 88795} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': 'd067d07a5100455dfd827110163dae49926829857adeb39ac76302ad9b064625', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.50', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 90603} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': 'cf3d651337c229a2ccf9b8b4d55e37a8d21f733262f1e6f64c0f789e42d81a3e', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 90628} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' CRITICAL:root:Exit data: {'ID': '0c64c49eb6d4ace2a3f31bbdcb86028732e8d3932ec776205daf2dd47c620794', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'cqlsh', 'arguments': ['10.0.0.50', '9041', '-e', 'quit'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 92651} WARNING:contrail-config-database-nodemgr:CQLSH failed, but Cassandra native transport port is open CRITICAL:root:Exit data: {'ID': '4a85c50e32d1efb92c3ad75863ad088e19b0a54a5e65bd1b269fcdb6fbb1974c', 'Running': False, 'ExitCode': 1, 'ProcessConfig': {'tty': True, 'entrypoint': 'nodetool', 'arguments': ['-p', '7201', 'compactionstats'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'c40306f447fefcd51cc0266baa470624c2ec914b5656244e9e5dc3f83a7cd112', 'DetachKeys': '', 'Pid': 92670} ERROR:contrail-config-database-nodemgr:Failed to get nodetool compactionstats: Result: 1 ERROR:contrail-config-database-nodemgr:Failed to get core files: [Errno 2] No such file or directory: '/var/crashes' [Introspect:8112]10.0.0.50 - - [2025-09-08 01:52:10] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 6735 0.001145