11/07/2024 02:25:44.004 7fb8bb71f808 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:25:44.005 7fb8bb71f808 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:25:44.005 7fb8bb71f808 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:25:44.005 7fb8b2cb06d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:25:44.006 7fb8b2cb06d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:25:44.006 7fb8b2cb06d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:25:44.007 7fb8b2cb06d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:25:44.007 7fb8b2cb06d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.007 7fb8b2cb06d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:25:44.007 7fb8b2cb06d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.007 7fb8b2cb06d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.007 7fb8b2cb06d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 start_time = 1730946344006882 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:25:44.007 7fb8b2cb06d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.075 7fb8bb71f808 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//'] 11/07/2024 02:25:44.087 7fb8bb71f808 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:25:44.088 7fb8b2c80630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.088 7fb8b2c80630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:25:44.089 7fb8b2c80630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.089 7fb8b2c80630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.089 7fb8b2c80630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:25:44.089 7fb8b2c80630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.092 7fb8bb71f808 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:25:44.092 7fb8b2c46eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.092 7fb8b2c46eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:25:44.093 7fb8b2c46eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.106 7fb8bb71f808 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:25:44.107 7fb8b2c39eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.107 7fb8b2c39eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:25:44.107 7fb8b2c39eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.110 7fb8bb71f808 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:25:44.112 7fb8b2c39eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.112 7fb8b2c39eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:25:44.112 7fb8b2c39eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.119 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:25:44.120 7fb8bb71f808 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:25:44.120 7fb8bb71f808 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:25:44.121 7fb8bb71f808 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:25:44.121 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:25:44.121 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:25:44.121 7fb8b2c46f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:25:44.122 7fb8b2c46f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:25:44.122 7fb8b2c39390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.122 7fb8b2c39390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:25:44.122 7fb8b2c39390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.123 7fb8bb71f808 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:25:44.127 7fb8b859e780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:25:44.127 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:25:44.128 7fb8b2c46f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:25:44.148 7fb8bb71f808 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:25:44.150 7fb8b29c96d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.150 7fb8b29c96d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:25:44.150 7fb8b29c96d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.157 7fb8bb71f808 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:25:44.157 7fb8b2a0f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.157 7fb8b2a0f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:25:44.157 7fb8b2a0f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.162 7fb8bb71f808 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:25:44.172 7fb8bb71f808 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:25:44.269 7fb8bb71f808 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/07/2024 02:25:44.446 7fb8bb71f808 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 11/07/2024 02:25:44.451 7fb8bb71f808 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/07/2024 02:25:44.570 7fb8bb71f808 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/07/2024 02:25:44.645 7fb8bb71f808 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 11/07/2024 02:25:44.651 7fb8bb71f808 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/07/2024 02:25:44.651 7fb8b2ac3f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.652 7fb8b2ac3f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:25:44.652 7fb8b2ac3f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.660 7fb8bb71f808 [contrail-device-manager] [ERROR]: Error in db walk read 'type' 11/07/2024 02:25:44.662 7fb8b29d2198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.662 7fb8b29d2198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:25:44.663 7fb8b29d2198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.671 7fb8b29d2ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.672 7fb8b29d2ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:25:44.672 7fb8b29d2ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:44.672 7fb8bb71f808 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:25:44.672 7fb8bb71f808 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:25:44.672 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:25:44.673 7fb8bb71f808 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1. 11/07/2024 02:25:44.676 7fb8b859e780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:25:44.686 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.'] 11/07/2024 02:25:44.696 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:25:44.700 7fb8b859e780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:25:44.705 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.'] 11/07/2024 02:25:45.187 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:25:45.707 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:46.254 7fb8bb71f808 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:25:46.483 7fb8bb71f808 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue 11/07/2024 02:25:46.483 7fb8bb71f808 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue 11/07/2024 02:25:46.791 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:46.813 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:25:46.822 7fb8b859e780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:25:46.833 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue'] 11/07/2024 02:25:46.984 7fb8bb71f808 [contrail-device-manager] [WARNING]: Process 114 prepared to run in Full mode... 11/07/2024 02:25:46.984 7fb8bb71f808 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:25:46.984 7fb8bb71f808 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:25:46.990 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:25:46.990 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:25:46.991 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:25:46.991 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:25:46.993 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:25:46.993 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:25:46.998 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:25:46.998 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:25:46.998 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:25:46.998 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:25:46.998 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:25:46.998 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:25:46.998 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:25:46.998 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:25:46.998 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:25:46.998 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:25:46.999 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:25:46.999 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:25:46.999 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:25:47.000 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:25:47.000 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:25:47.000 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:25:47.001 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:25:47.001 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:25:47.001 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:25:47.001 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:25:47.002 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:25:47.002 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:25:47.002 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:25:47.002 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:25:47.002 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:25:47.002 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:25:47.003 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:25:47.003 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:25:47.003 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:25:47.003 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:25:47.003 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:25:47.003 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:25:47.004 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:25:47.004 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:25:47.004 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:25:47.004 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:25:47.004 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:25:47.004 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:25:47.004 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:25:47.004 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:25:47.005 7fb8bb71f808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:25:47.008 7fb8bb71f808 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:25:47.073 7fb8b2903828 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:25:47.073 7fb8b2903828 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:25:47.073 7fb8b20f60f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:47.073 7fb8b20f60f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:25:47.073 7fb8b20f60f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:47.079 7fb8b2814588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:47.079 7fb8b2814588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:25:47.079 7fb8b2814588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:47.836 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:47.843 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:25:47.847 7fb8b859e780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:25:47.859 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:25:48.007 7fb8b20284e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:25:48.008 7fb8b20284e0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:25:48.008 7fb8b20284e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:25:48.008 7fb8b20284e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:48.009 7fb8b20284e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:25:48.009 7fb8b20284e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:48.009 7fb8b20284e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:25:48.009 7fb8b20284e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:25:48.009 7fb8b20284e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:48.009 7fb8b20284e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:25:48.009 7fb8b20284e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:48.009 7fb8b20284e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:48.010 7fb8b20284e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:25:48.010 7fb8b20284e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:48.862 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:49.864 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:50.193 7fb8b20254e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:50.193 7fb8b20254e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:25:50.193 7fb8b20254e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:50.865 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:51.867 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:52.009 7fb8b1782cc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:25:52.010 7fb8b1782cc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:25:52.010 7fb8b1782cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:25:52.010 7fb8b1782cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:52.010 7fb8b1782cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:25:52.010 7fb8b1782cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:52.010 7fb8b1782cc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:25:52.011 7fb8b1782cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:25:52.011 7fb8b1782cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:52.011 7fb8b1782cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:25:52.011 7fb8b1782cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:52.011 7fb8b1782cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:52.011 7fb8b1782cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:25:52.011 7fb8b1782cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:52.868 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:53.869 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:54.870 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:55.871 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:56.010 7fb8b16d8780 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:25:56.011 7fb8b16d8780 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:25:56.011 7fb8b16d8780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:25:56.011 7fb8b16d8780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:56.012 7fb8b16d8780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:25:56.012 7fb8b16d8780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:56.012 7fb8b16d8780 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:25:56.012 7fb8b16d8780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:25:56.012 7fb8b16d8780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:56.013 7fb8b16d8780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:25:56.013 7fb8b16d8780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:56.013 7fb8b16d8780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:56.013 7fb8b16d8780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:25:56.013 7fb8b16d8780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:56.872 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:57.874 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:58.874 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:59.875 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:00.011 7fb8b16d8048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:00.012 7fb8b16d8048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:00.013 7fb8b16d8048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:00.013 7fb8b16d8048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:00.013 7fb8b16d8048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:00.013 7fb8b16d8048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:00.013 7fb8b16d8048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:00.013 7fb8b16d8048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:00.013 7fb8b16d8048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.014 7fb8b16d8048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:00.014 7fb8b16d8048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.014 7fb8b16d8048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.014 7fb8b16d8048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:00.014 7fb8b16d8048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.876 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:01.877 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:02.878 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:03.879 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:04.013 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:04.014 7fb8b2ac3eb8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:04.014 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:04.015 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:04.015 7fb8b2ac3eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:04.015 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:04.015 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:04.015 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:04.015 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:04.015 7fb8b2ac3eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:04.016 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:04.016 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:04.016 7fb8b2ac3eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:04.016 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:04.880 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:05.881 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:06.882 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:07.883 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:08.015 7fb8b1751438 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:08.016 7fb8b1751438 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:08.016 7fb8b1751438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:08.016 7fb8b1751438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:08.017 7fb8b1751438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:08.017 7fb8b1751438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:08.017 7fb8b1751438 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:08.017 7fb8b1751438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:08.017 7fb8b1751438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:08.017 7fb8b1751438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:08.017 7fb8b1751438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:08.018 7fb8b1751438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:08.018 7fb8b1751438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:08.018 7fb8b1751438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:08.884 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:09.885 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:10.886 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:11.887 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:12.017 7fb8b17530f0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:12.018 7fb8b17530f0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:12.018 7fb8b17530f0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:12.018 7fb8b17530f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:12.018 7fb8b17530f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:12.018 7fb8b17530f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:12.018 7fb8b17530f0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:12.019 7fb8b17530f0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:12.019 7fb8b17530f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:12.019 7fb8b17530f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:12.019 7fb8b17530f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:12.019 7fb8b17530f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:12.019 7fb8b17530f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:12.019 7fb8b17530f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:12.888 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:13.889 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:14.890 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:15.891 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:16.018 7fb8b1753588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:16.020 7fb8b1753588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:16.020 7fb8b1753588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:16.020 7fb8b1753588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:16.020 7fb8b1753588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:16.020 7fb8b1753588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:16.020 7fb8b1753588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:16.021 7fb8b1753588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:16.021 7fb8b1753588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:16.021 7fb8b1753588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:16.021 7fb8b1753588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:16.021 7fb8b1753588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:16.021 7fb8b1753588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:16.021 7fb8b1753588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:16.892 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:17.893 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:18.895 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:19.896 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:20.020 7fb8b201e048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:20.022 7fb8b201e048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:20.022 7fb8b201e048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:20.022 7fb8b201e048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:20.022 7fb8b201e048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:20.022 7fb8b201e048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:20.022 7fb8b201e048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:20.023 7fb8b201e048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:20.023 7fb8b201e048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:20.023 7fb8b201e048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:20.023 7fb8b201e048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:20.023 7fb8b201e048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:20.023 7fb8b201e048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:20.023 7fb8b201e048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:20.897 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:21.899 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:22.900 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:23.901 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:24.022 7fb8ba454cc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:24.023 7fb8ba454cc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:24.023 7fb8ba454cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:24.023 7fb8ba454cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:24.024 7fb8ba454cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:24.024 7fb8ba454cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:24.024 7fb8ba454cc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:24.024 7fb8ba454cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:24.024 7fb8ba454cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:24.024 7fb8ba454cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:24.025 7fb8ba454cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:24.025 7fb8ba454cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:24.025 7fb8ba454cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:24.025 7fb8ba454cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:24.902 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:25.903 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:26.904 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:27.905 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:28.024 7fb8b201e048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:28.027 7fb8b201e048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:28.028 7fb8b201e048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:28.028 7fb8b201e048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:28.028 7fb8b201e048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:28.028 7fb8b201e048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:28.028 7fb8b201e048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:28.028 7fb8b201e048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:28.029 7fb8b201e048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:28.029 7fb8b201e048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:28.029 7fb8b201e048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:28.029 7fb8b201e048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:28.029 7fb8b201e048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:28.029 7fb8b201e048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:28.906 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:29.907 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:30.908 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:31.909 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:32.028 7fb8b17538d0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:32.029 7fb8b17538d0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:32.029 7fb8b17538d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:32.030 7fb8b17538d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:32.030 7fb8b17538d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:32.030 7fb8b17538d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:32.030 7fb8b17538d0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:32.030 7fb8b17538d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:32.030 7fb8b17538d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:32.030 7fb8b17538d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:32.031 7fb8b17538d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:32.031 7fb8b17538d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:32.031 7fb8b17538d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:32.031 7fb8b17538d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:32.910 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:33.911 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:34.912 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:35.914 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:36.030 7fb8b201e240 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:36.031 7fb8b201e240 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:36.031 7fb8b201e240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:36.031 7fb8b201e240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:36.032 7fb8b201e240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:36.032 7fb8b201e240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:36.032 7fb8b201e240 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:36.032 7fb8b201e240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:36.032 7fb8b201e240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:36.032 7fb8b201e240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:36.032 7fb8b201e240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:36.032 7fb8b201e240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:36.033 7fb8b201e240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:36.033 7fb8b201e240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:36.915 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:37.915 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:38.917 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:39.917 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:40.032 7fb8b20416d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:40.035 7fb8b20416d8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:40.035 7fb8b20416d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:40.035 7fb8b20416d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:40.036 7fb8b20416d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:40.036 7fb8b20416d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:40.036 7fb8b20416d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:40.036 7fb8b20416d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:40.036 7fb8b20416d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:40.036 7fb8b20416d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:40.036 7fb8b20416d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:40.036 7fb8b20416d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:40.037 7fb8b20416d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:40.037 7fb8b20416d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:40.918 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:41.919 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:42.921 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:43.921 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:44.035 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:44.036 7fb8b2ac3eb8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:44.037 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:44.037 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:44.037 7fb8b2ac3eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:44.037 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:44.037 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:44.038 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:44.038 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:44.038 7fb8b2ac3eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:44.038 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:44.038 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:44.038 7fb8b2ac3eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:44.038 7fb8b2ac3eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:44.923 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:45.923 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:46.924 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:47.926 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:48.037 7fb8b1749ac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:48.038 7fb8b1749ac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:48.038 7fb8b1749ac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:48.038 7fb8b1749ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:48.038 7fb8b1749ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:48.038 7fb8b1749ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:48.038 7fb8b1749ac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:48.039 7fb8b1749ac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:48.039 7fb8b1749ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:48.039 7fb8b1749ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:48.039 7fb8b1749ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:48.039 7fb8b1749ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:48.039 7fb8b1749ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:48.039 7fb8b1749ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:48.927 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:49.928 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:50.928 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:51.929 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:52.039 7fb8b1756588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:52.040 7fb8b1756588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:52.040 7fb8b1756588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:52.040 7fb8b1756588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:52.040 7fb8b1756588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:52.040 7fb8b1756588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:52.040 7fb8b1756588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:52.041 7fb8b1756588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:52.041 7fb8b1756588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:52.041 7fb8b1756588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:52.041 7fb8b1756588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:52.041 7fb8b1756588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:52.041 7fb8b1756588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946344006882 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:52.041 7fb8b1756588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:52.930 7fb8b859e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:42.729 7fdd7eb3c208 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:27:42.730 7fdd7eb3c208 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:27:42.730 7fdd7eb3c208 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:27:42.730 7fdd761d46d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:27:42.730 7fdd761d46d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:27:42.731 7fdd761d46d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:27:42.731 7fdd761d46d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:27:42.731 7fdd761d46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.731 7fdd761d46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:42.731 7fdd761d46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.731 7fdd761d46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.731 7fdd761d46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 start_time = 1730946462731040 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:27:42.731 7fdd761d46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.769 7fdd7eb3c208 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//'] 11/07/2024 02:27:42.774 7fdd7eb3c208 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:27:42.775 7fdd76122240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.775 7fdd76122240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:27:42.775 7fdd76122240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.775 7fdd76122240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.775 7fdd76122240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:27:42.775 7fdd76122240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.778 7fdd7eb3c208 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:27:42.778 7fdd76122240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.779 7fdd76122240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:42.779 7fdd76122240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.785 7fdd7eb3c208 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:27:42.785 7fdd7607b2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.785 7fdd7607b2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:27:42.785 7fdd7607b2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.788 7fdd7eb3c208 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:27:42.788 7fdd7607b2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.788 7fdd7607b2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:42.788 7fdd7607b2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.796 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:27:42.796 7fdd7eb3c208 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:27:42.796 7fdd7eb3c208 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:27:42.796 7fdd7eb3c208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:27:42.797 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:27:42.797 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:42.797 7fdd78be40f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:27:42.797 7fdd78be40f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:27:42.798 7fdd7607bcc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.798 7fdd7607bcc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:27:42.798 7fdd7607bcc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.799 7fdd7eb3c208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:27:42.802 7fdd76055e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:42.802 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:27:42.802 7fdd78be40f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:27:42.821 7fdd7eb3c208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:27:42.823 7fdd7571b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.823 7fdd7571b8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:27:42.823 7fdd7571b8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.827 7fdd7eb3c208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:27:42.828 7fdd757c0828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.828 7fdd757c0828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:27:42.828 7fdd757c0828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.831 7fdd7eb3c208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:27:42.838 7fdd7eb3c208 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:27:42.839 7fdd757552e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.839 7fdd757552e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:42.839 7fdd757552e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.853 7fdd760210f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.854 7fdd760210f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:27:42.854 7fdd760210f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.860 7fdd7607b630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.860 7fdd7607b630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:42.860 7fdd7607b630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:42.860 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:27:42.860 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:27:42.860 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:42.861 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1. 11/07/2024 02:27:42.863 7fdd76055e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:42.869 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.'] 11/07/2024 02:27:42.875 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:42.878 7fdd76055e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:42.883 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.'] 11/07/2024 02:27:43.373 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:27:43.550 7fdd7eb3c208 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:27:43.722 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue 11/07/2024 02:27:43.722 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue 11/07/2024 02:27:43.885 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:43.891 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:43.895 7fdd76055e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:43.902 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue'] 11/07/2024 02:27:44.223 7fdd7eb3c208 [contrail-device-manager] [WARNING]: Process 114 prepared to run in Full mode... 11/07/2024 02:27:44.223 7fdd7eb3c208 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:27:44.223 7fdd7eb3c208 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:27:44.227 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:27:44.227 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:27:44.228 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:27:44.228 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:27:44.229 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:27:44.229 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:27:44.234 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:27:44.234 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:27:44.234 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:27:44.234 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:27:44.234 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:27:44.234 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:27:44.234 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:27:44.234 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:27:44.234 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:27:44.235 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:27:44.235 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:27:44.235 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:27:44.235 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:27:44.236 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:27:44.237 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:27:44.237 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:27:44.237 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:27:44.237 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:27:44.237 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:27:44.237 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:27:44.238 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:27:44.238 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:27:44.238 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:27:44.238 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:27:44.238 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:27:44.238 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:27:44.239 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:27:44.239 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:27:44.240 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:27:44.240 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:27:44.240 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:27:44.240 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:27:44.240 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:27:44.240 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:27:44.240 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:27:44.240 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:27:44.240 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:27:44.241 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:27:44.241 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:27:44.241 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:27:44.241 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:27:44.244 7fdd7eb3c208 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:27:44.291 7fdd75676630 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:27:44.291 7fdd75676630 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:27:44.291 7fdd756d3438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:44.292 7fdd756d3438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:27:44.292 7fdd756d3438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:44.308 7fdd756c1780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:44.308 7fdd756c1780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:44.308 7fdd756c1780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:44.904 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:44.912 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:44.916 7fdd76055e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:44.928 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:27:45.930 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:46.731 7fdd74ca1eb8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:27:46.732 7fdd74ca1eb8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:27:46.732 7fdd74ca1eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:27:46.733 7fdd74ca1eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:27:46.733 7fdd74ca1eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:27:46.733 7fdd74ca1eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:27:46.733 7fdd74ca1eb8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:27:46.733 7fdd74ca1eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:27:46.733 7fdd74ca1eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:46.734 7fdd74ca1eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:27:46.734 7fdd74ca1eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:46.734 7fdd74ca1eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:46.734 7fdd74ca1eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:27:46.734 7fdd74ca1eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:46.931 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:47.933 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:48.934 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:49.935 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:50.735 7fdd74ca1f60 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:27:50.736 7fdd74ca1f60 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:27:50.736 7fdd74ca1f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:27:50.736 7fdd74ca1f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:27:50.736 7fdd74ca1f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:27:50.736 7fdd74ca1f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:27:50.736 7fdd74ca1f60 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:27:50.737 7fdd74ca1f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:27:50.737 7fdd74ca1f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:50.737 7fdd74ca1f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:27:50.737 7fdd74ca1f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:50.737 7fdd74ca1f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:50.737 7fdd74ca1f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:27:50.737 7fdd74ca1f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:50.936 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:51.938 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:52.939 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:53.940 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:54.736 7fdd74cae198 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:27:54.739 7fdd74cae198 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:27:54.740 7fdd74cae198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:27:54.740 7fdd74cae198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:27:54.740 7fdd74cae198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:27:54.740 7fdd74cae198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:27:54.740 7fdd74cae198 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:27:54.740 7fdd74cae198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:27:54.741 7fdd74cae198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:54.741 7fdd74cae198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:27:54.741 7fdd74cae198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:54.741 7fdd74cae198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:54.741 7fdd74cae198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:27:54.741 7fdd74cae198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:54.941 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:55.942 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:56.944 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:57.945 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:58.740 7fdd74cae240 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:27:58.741 7fdd74cae240 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:27:58.741 7fdd74cae240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:27:58.741 7fdd74cae240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:27:58.742 7fdd74cae240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:27:58.742 7fdd74cae240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:27:58.742 7fdd74cae240 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:27:58.742 7fdd74cae240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:27:58.742 7fdd74cae240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:58.742 7fdd74cae240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:27:58.742 7fdd74cae240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:58.742 7fdd74cae240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:58.743 7fdd74cae240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:27:58.743 7fdd74cae240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:58.946 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:59.947 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:00.948 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:01.949 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:02.742 7fdd74ca1d68 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:28:02.743 7fdd74ca1d68 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:28:02.743 7fdd74ca1d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:28:02.743 7fdd74ca1d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:02.743 7fdd74ca1d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:28:02.744 7fdd74ca1d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:02.744 7fdd74ca1d68 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:28:02.744 7fdd74ca1d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:28:02.744 7fdd74ca1d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:02.744 7fdd74ca1d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:28:02.744 7fdd74ca1d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:02.744 7fdd74ca1d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:02.744 7fdd74ca1d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:28:02.745 7fdd74ca1d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:02.950 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:03.952 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:04.953 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:05.953 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:06.744 7fdd7600b438 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:28:06.745 7fdd7600b438 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:28:06.745 7fdd7600b438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:28:06.746 7fdd7600b438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:06.746 7fdd7600b438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:28:06.746 7fdd7600b438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:06.746 7fdd7600b438 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:28:06.746 7fdd7600b438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:28:06.746 7fdd7600b438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:06.746 7fdd7600b438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:28:06.747 7fdd7600b438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:06.747 7fdd7600b438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:06.747 7fdd7600b438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:28:06.747 7fdd7600b438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:06.954 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:07.955 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:08.955 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:09.956 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:10.746 7fdd74caec18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:28:10.747 7fdd74caec18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:28:10.747 7fdd74caec18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:28:10.747 7fdd74caec18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:10.748 7fdd74caec18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:28:10.748 7fdd74caec18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:10.748 7fdd74caec18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:28:10.748 7fdd74caec18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:28:10.748 7fdd74caec18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:10.748 7fdd74caec18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:28:10.748 7fdd74caec18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:10.748 7fdd74caec18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:10.749 7fdd74caec18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:28:10.749 7fdd74caec18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:10.957 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:11.958 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:12.959 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:13.961 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:14.748 7fdd74caee10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:28:14.749 7fdd74caee10 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:28:14.749 7fdd74caee10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:28:14.749 7fdd74caee10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:14.749 7fdd74caee10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:28:14.750 7fdd74caee10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:14.750 7fdd74caee10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:28:14.750 7fdd74caee10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:28:14.750 7fdd74caee10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:14.750 7fdd74caee10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:28:14.750 7fdd74caee10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:14.750 7fdd74caee10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:14.750 7fdd74caee10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:28:14.751 7fdd74caee10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:14.961 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:15.963 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:16.964 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:17.965 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:18.749 7fdd74caec18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:28:18.750 7fdd74caec18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:28:18.750 7fdd74caec18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:28:18.751 7fdd74caec18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:18.751 7fdd74caec18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:28:18.751 7fdd74caec18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:18.751 7fdd74caec18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:28:18.751 7fdd74caec18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:28:18.751 7fdd74caec18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:18.751 7fdd74caec18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:28:18.752 7fdd74caec18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:18.752 7fdd74caec18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:18.752 7fdd74caec18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:28:18.752 7fdd74caec18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:18.966 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:19.966 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:20.967 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:21.968 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:22.751 7fdd74cbd780 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:28:22.752 7fdd74cbd780 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:28:22.752 7fdd74cbd780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:28:22.752 7fdd74cbd780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:22.752 7fdd74cbd780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:28:22.753 7fdd74cbd780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:22.753 7fdd74cbd780 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:28:22.753 7fdd74cbd780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:28:22.753 7fdd74cbd780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:22.753 7fdd74cbd780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:28:22.753 7fdd74cbd780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:22.753 7fdd74cbd780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:22.753 7fdd74cbd780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:28:22.754 7fdd74cbd780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:22.969 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:23.970 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:24.971 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:25.973 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:26.752 7fdd74cbd8d0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:28:26.754 7fdd74cbd8d0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:28:26.754 7fdd74cbd8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:28:26.754 7fdd74cbd8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:26.754 7fdd74cbd8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:28:26.754 7fdd74cbd8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:26.754 7fdd74cbd8d0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:28:26.755 7fdd74cbd8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:28:26.755 7fdd74cbd8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:26.755 7fdd74cbd8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:28:26.755 7fdd74cbd8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:26.755 7fdd74cbd8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:26.755 7fdd74cbd8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:28:26.755 7fdd74cbd8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:26.973 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:27.974 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:28.975 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:29.976 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:30.754 7fdd74cbd780 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:28:30.755 7fdd74cbd780 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:28:30.756 7fdd74cbd780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:28:30.756 7fdd74cbd780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:30.756 7fdd74cbd780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:28:30.756 7fdd74cbd780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:30.756 7fdd74cbd780 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:28:30.756 7fdd74cbd780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:28:30.756 7fdd74cbd780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:30.757 7fdd74cbd780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:28:30.757 7fdd74cbd780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:30.757 7fdd74cbd780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:30.757 7fdd74cbd780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:28:30.757 7fdd74cbd780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:30.977 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:31.979 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:32.980 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:33.982 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:34.756 7fdd74d27d68 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:28:34.758 7fdd74d27d68 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:28:34.758 7fdd74d27d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:28:34.758 7fdd74d27d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:34.758 7fdd74d27d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:28:34.758 7fdd74d27d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:34.758 7fdd74d27d68 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:28:34.759 7fdd74d27d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:28:34.759 7fdd74d27d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:34.759 7fdd74d27d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:28:34.759 7fdd74d27d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:34.759 7fdd74d27d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:34.759 7fdd74d27d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:28:34.759 7fdd74d27d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:34.982 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:35.984 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:36.984 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:37.986 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:38.759 7fdd74cae198 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:28:38.760 7fdd74cae198 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:28:38.760 7fdd74cae198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:28:38.760 7fdd74cae198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:38.760 7fdd74cae198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:28:38.760 7fdd74cae198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:38.760 7fdd74cae198 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:28:38.761 7fdd74cae198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:28:38.761 7fdd74cae198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:38.761 7fdd74cae198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:28:38.761 7fdd74cae198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:38.761 7fdd74cae198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:38.761 7fdd74cae198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:28:38.761 7fdd74cae198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:38.986 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:39.987 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:40.988 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:41.989 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:42.760 7fdd74caee10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:28:42.764 7fdd74caee10 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:28:42.764 7fdd74caee10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:28:42.764 7fdd74caee10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:42.764 7fdd74caee10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:28:42.764 7fdd74caee10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:42.764 7fdd74caee10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:28:42.765 7fdd74caee10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:28:42.765 7fdd74caee10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:42.765 7fdd74caee10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:28:42.765 7fdd74caee10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:42.765 7fdd74caee10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:42.766 7fdd74caee10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946462731040 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:28:42.766 7fdd74caee10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:42.990 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:43.616 7fdd7eb3c208 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: adb9bfdc-df7a-48d0-b908-785940bb6a85 11/07/2024 02:28:43.772 7fdd7eb3c208 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: f1cff9ac-2ab7-4b58-8390-d193a819273b 11/07/2024 02:28:43.991 7fdd76055e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:28.772 7f6981c57f48 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:29:28.772 7f6981c57f48 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:29:28.772 7f6981c57f48 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:29:28.772 7f69792cb6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:29:28.774 7f69792cb6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:29:28.774 7f69792cb6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:29:28.774 7f69792cb6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:29:28.775 7f69792cb6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.775 7f69792cb6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:28.775 7f69792cb6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.775 7f69792cb6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.776 7f69792cb6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 113 start_time = 1730946568774724 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:29:28.776 7f69792cb6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.816 7f6981c57f48 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//'] 11/07/2024 02:29:28.821 7f6981c57f48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:29:28.821 7f697921a240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.822 7f697921a240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:29:28.822 7f697921a240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.822 7f697921a240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.822 7f697921a240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:29:28.822 7f697921a240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.825 7f6981c57f48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:29:28.825 7f697921a0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.825 7f697921a0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:28.825 7f697921a0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.832 7f6981c57f48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:29:28.832 7f69791732e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.832 7f69791732e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:29:28.834 7f69791732e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.836 7f6981c57f48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:29:28.837 7f69791732e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.837 7f69791732e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:28.837 7f69791732e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.843 7f6981c57f48 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:29:28.843 7f6981c57f48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:29:28.844 7f6981c57f48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:29:28.844 7f6981c57f48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:29:28.844 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:29:28.844 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:28.845 7f697bcdf0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:29:28.845 7f697bcdf0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:29:28.845 7f6979173cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.845 7f6979173cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:29:28.845 7f6979173cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.846 7f6981c57f48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:29:28.850 7f69818a1e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:28.850 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:29:28.851 7f697bcdf0f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:29:28.888 7f6981c57f48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:29:28.889 7f6979017780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.889 7f6979017780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:29:28.889 7f6979017780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.895 7f6981c57f48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:29:28.896 7f6979019c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.897 7f6979019c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:29:28.897 7f6979019c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.902 7f6981c57f48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:29:28.917 7f6981c57f48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:29:28.918 7f697902a6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.919 7f697902a6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:28.919 7f697902a6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.934 7f6979120f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.934 7f6979120f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:29:28.934 7f6979120f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.945 7f69802cc828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.946 7f69802cc828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:28.946 7f69802cc828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:28.946 7f6981c57f48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:29:28.946 7f6981c57f48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:29:28.946 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:28.947 7f6981c57f48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1. 11/07/2024 02:29:28.954 7f69818a1e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:28.962 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.'] 11/07/2024 02:29:28.975 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:28.979 7f69818a1e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:28.987 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.'] 11/07/2024 02:29:29.460 7f6981c57f48 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:29:29.730 7f6981c57f48 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:29:29.945 7f6981c57f48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue 11/07/2024 02:29:29.945 7f6981c57f48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue 11/07/2024 02:29:29.990 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:29.997 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:30.002 7f69818a1e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:30.011 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue'] 11/07/2024 02:29:30.447 7f6981c57f48 [contrail-device-manager] [WARNING]: Process 113 prepared to run in Full mode... 11/07/2024 02:29:30.447 7f6981c57f48 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:29:30.447 7f6981c57f48 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:29:30.451 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:29:30.451 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:29:30.452 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:29:30.452 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:29:30.453 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:29:30.453 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:29:30.458 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:29:30.459 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:29:30.459 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:29:30.459 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:29:30.459 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:29:30.459 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:29:30.459 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:29:30.459 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:29:30.459 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:29:30.460 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:29:30.460 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:29:30.460 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:29:30.460 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:29:30.461 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:29:30.462 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:29:30.462 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:29:30.462 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:29:30.462 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:29:30.462 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:29:30.463 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:29:30.463 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:29:30.464 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:29:30.464 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:29:30.464 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:29:30.465 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:29:30.465 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:29:30.466 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:29:30.466 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:29:30.466 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:29:30.466 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:29:30.466 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:29:30.466 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:29:30.466 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:29:30.466 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:29:30.467 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:29:30.467 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:29:30.467 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:29:30.467 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:29:30.467 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:29:30.467 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:29:30.467 7f6981c57f48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:29:30.470 7f6981c57f48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:29:30.518 7f6978e8f828 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:29:30.519 7f6978e8f828 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:29:30.519 7f6978d66f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:30.519 7f6978d66f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:29:30.519 7f6978d66f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:30.536 7f6978e2a4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:30.536 7f6978e2a4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:30.536 7f6978e2a4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:31.014 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:31.021 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:31.026 7f69818a1e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:31.039 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:29:32.043 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:32.774 7f6978591eb8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:29:32.776 7f6978591eb8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:29:32.776 7f6978591eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:29:32.776 7f6978591eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:29:32.777 7f6978591eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:29:32.777 7f6978591eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:29:32.777 7f6978591eb8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:29:32.777 7f6978591eb8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:29:32.777 7f6978591eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:29:32.777 7f6978591eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:29:32.777 7f6978591eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:29:32.778 7f69785a3438 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:29:32.779 7f69785a3438 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:29:32.782 7f69785a32e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 11/07/2024 02:29:32.784 7f69785a32e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:29:32.784 7f6979e77eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:29:32.785 7f6979e77eb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:29:32.785 7f6979e77eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:29:32.785 7f6979e77eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:29:32.785 7f6979e77eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:29:32.785 7f69785a3048 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:29:32.787 7f69785a3048 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:29:33.045 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:34.046 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:35.048 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:36.133 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:37.134 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:38.135 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:39.136 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:40.138 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:41.139 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:42.140 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:43.142 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:44.142 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:45.144 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:46.145 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:47.146 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:48.148 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:49.149 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:50.150 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:51.152 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:52.153 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:53.153 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:54.155 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:55.156 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:56.157 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:57.158 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:58.159 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:59.160 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:00.161 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:01.162 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:02.164 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:03.165 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:04.166 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:05.167 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:06.168 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:07.169 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:08.170 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:09.172 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:10.173 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:11.174 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:12.175 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:13.176 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:14.176 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:15.177 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:16.178 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:17.179 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:18.180 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:19.181 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:20.182 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:21.183 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:22.185 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:23.186 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:24.187 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:25.188 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:26.189 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:27.190 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:28.192 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:29.194 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:30.194 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:31.195 7f69818a1e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:31.452 7f6981c57f48 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: adb9bfdc-df7a-48d0-b908-785940bb6a85 11/07/2024 02:30:31.608 7f6981c57f48 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: f1cff9ac-2ab7-4b58-8390-d193a819273b 11/07/2024 02:31:17.506 7fd70755cb08 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:31:17.506 7fd70755cb08 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:31:17.506 7fd70755cb08 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:31:17.507 7fd6fec2a6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:31:17.507 7fd6fec2a6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:31:17.508 7fd6fec2a6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:31:17.508 7fd6fec2a6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:31:17.508 7fd6fec2a6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.508 7fd6fec2a6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:17.508 7fd6fec2a6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.508 7fd6fec2a6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.509 7fd6fec2a6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 113 start_time = 1730946677508111 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:31:17.509 7fd6fec2a6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.582 7fd70755cb08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//'] 11/07/2024 02:31:17.593 7fd70755cb08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:31:17.594 7fd6feb7a240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.594 7fd6feb7a240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:31:17.595 7fd6feb7a240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.595 7fd6feb7a240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.595 7fd6feb7a240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:31:17.595 7fd6feb7a240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.599 7fd70755cb08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:31:17.600 7fd6feb7a0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.600 7fd6feb7a0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:17.600 7fd6feb7a0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.609 7fd70755cb08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:31:17.610 7fd6fead22e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.610 7fd6fead22e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:31:17.610 7fd6fead22e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.613 7fd70755cb08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:31:17.613 7fd6fead22e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.613 7fd6fead22e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:17.614 7fd6fead22e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.621 7fd70755cb08 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:31:17.621 7fd70755cb08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:31:17.621 7fd70755cb08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:31:17.622 7fd70755cb08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:31:17.622 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:31:17.622 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:31:17.623 7fd70163d0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:31:17.623 7fd70163d0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:31:17.623 7fd6fead2cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.623 7fd6fead2cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:31:17.624 7fd6fead2cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.625 7fd70755cb08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:31:17.637 7fd707205d68 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:31:17.637 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:31:17.638 7fd70163d0f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:31:17.650 7fd70755cb08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:31:17.652 7fd6fe977780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.652 7fd6fe977780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:31:17.652 7fd6fe977780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.658 7fd70755cb08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:31:17.658 7fd6fe97ac18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.658 7fd6fe97ac18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:31:17.658 7fd6fe97ac18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.663 7fd70755cb08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:31:17.671 7fd70755cb08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:31:17.672 7fd6fe98b6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.672 7fd6fe98b6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:17.672 7fd6fe98b6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.686 7fd6fea79b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.686 7fd6fea79b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:31:17.686 7fd6fea79b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.694 7fd6fea62d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.694 7fd6fea62d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:17.694 7fd6fea62d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:17.695 7fd70755cb08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:31:17.695 7fd70755cb08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:31:17.695 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:31:17.695 7fd70755cb08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1. 11/07/2024 02:31:17.699 7fd707205d68 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:31:17.705 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.'] 11/07/2024 02:31:17.711 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:31:17.715 7fd707205d68 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:31:17.722 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.'] 11/07/2024 02:31:18.266 7fd70755cb08 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:31:18.724 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:18.766 7fd70755cb08 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:31:19.009 7fd70755cb08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue 11/07/2024 02:31:19.009 7fd70755cb08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue 11/07/2024 02:31:19.510 7fd70755cb08 [contrail-device-manager] [WARNING]: Process 113 prepared to run in Full mode... 11/07/2024 02:31:19.510 7fd70755cb08 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:31:19.511 7fd70755cb08 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:31:19.517 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:31:19.517 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:31:19.519 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:31:19.519 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:31:19.521 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:31:19.522 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:31:19.531 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:31:19.531 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:31:19.531 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:31:19.531 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:31:19.531 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:31:19.531 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:31:19.531 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:31:19.531 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:31:19.532 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:31:19.532 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:31:19.532 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:31:19.532 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:31:19.532 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:31:19.535 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:31:19.535 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:31:19.535 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:31:19.535 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:31:19.536 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:31:19.536 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:31:19.536 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:31:19.537 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:31:19.537 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:31:19.537 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:31:19.538 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:31:19.538 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:31:19.538 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:31:19.539 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:31:19.540 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:31:19.540 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:31:19.540 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:31:19.540 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:31:19.540 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:31:19.540 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:31:19.540 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:31:19.540 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:31:19.540 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:31:19.540 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:31:19.541 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:31:19.541 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:31:19.541 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:31:19.541 7fd70755cb08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:31:19.544 7fd70755cb08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:31:19.641 7fd6fe89a630 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:31:19.641 7fd6fe89a630 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:31:19.642 7fd6fe7bf8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.642 7fd6fe7bf8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:31:19.642 7fd6fe7bf8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.671 7fd6fe7b4390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.671 7fd6fe7b4390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:19.671 7fd6fe7b4390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.725 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:19.731 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:31:19.735 7fd707205d68 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:31:19.746 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:31:20.749 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:21.508 7fd6fd6f42e8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:31:21.509 7fd6fd6f42e8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:31:21.509 7fd6fd6f42e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:31:21.509 7fd6fd6f42e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:31:21.509 7fd6fd6f42e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:31:21.509 7fd6fd6f42e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:31:21.509 7fd6fd6f42e8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:31:21.510 7fd6fd6f42e8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:31:21.510 7fd6fd6f42e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:31:21.510 7fd6fd6f42e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:31:21.510 7fd6fd6f42e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:31:21.510 7fd6fd6f9ac8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:31:21.511 7fd6fd6f9ac8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:31:21.513 7fd6fd6f9cc0 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 11/07/2024 02:31:21.514 7fd6fd6f9cc0 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:31:21.515 7fd6fd6f42e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:31:21.515 7fd6fd6f42e8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:31:21.515 7fd6fd6f42e8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:31:21.515 7fd6fd6f42e8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:31:21.515 7fd6fd6f42e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:31:21.516 7fd6fd6f9c18 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:31:21.517 7fd6fd6f9c18 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:31:21.751 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:22.752 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:23.754 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:24.755 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:25.757 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:26.759 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:27.759 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:28.773 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:29.774 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:30.430 7fd6fd6f4f60 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:31:30.775 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:31.776 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:32.718 7fd6fdfa86d8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:31:32.777 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:33.777 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:34.778 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:35.780 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:36.780 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:37.782 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:38.784 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:39.784 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:40.785 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:41.787 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:42.788 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:43.789 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:44.789 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:45.791 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:46.792 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:47.792 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:48.793 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:49.795 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:50.796 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:51.797 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:52.798 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:53.799 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:54.800 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:55.801 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:56.802 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:57.803 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:58.804 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:59.805 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:00.806 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:01.807 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:02.809 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:03.810 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:04.812 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:05.812 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:06.814 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:07.815 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:08.816 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:09.816 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:10.817 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:11.819 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:12.820 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:13.822 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:14.823 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:15.824 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:16.825 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:17.826 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:18.827 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:19.540 7fd70755cb08 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: adb9bfdc-df7a-48d0-b908-785940bb6a85 11/07/2024 02:32:19.696 7fd70755cb08 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: f1cff9ac-2ab7-4b58-8390-d193a819273b 11/07/2024 02:32:19.828 7fd707205d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:02.856 7f6c077848c8 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:33:02.856 7f6c077848c8 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:33:02.856 7f6c077848c8 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:33:02.856 7f6bfedf46d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:33:02.856 7f6bfedf46d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:33:02.857 7f6bfedf46d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:33:02.857 7f6bfedf46d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:33:02.857 7f6bfedf46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.857 7f6bfedf46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:33:02.857 7f6bfedf46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.858 7f6bfedf46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.858 7f6bfedf46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 113 start_time = 1730946782857288 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:33:02.858 7f6bfedf46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.899 7f6c077848c8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//'] 11/07/2024 02:33:02.905 7f6c077848c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:33:02.905 7f6bfed43240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.906 7f6bfed43240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:33:02.906 7f6bfed43240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.906 7f6bfed43240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.906 7f6bfed43240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:33:02.906 7f6bfed43240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.909 7f6c077848c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:33:02.909 7f6bfed430f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.910 7f6bfed430f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:33:02.910 7f6bfed430f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.916 7f6c077848c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:33:02.916 7f6bfec9b2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.917 7f6bfec9b2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:33:02.917 7f6bfec9b2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.919 7f6c077848c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:33:02.920 7f6bfec9b240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.920 7f6bfec9b240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:33:02.920 7f6bfec9b240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.928 7f6c077848c8 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:33:02.928 7f6c077848c8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:33:02.928 7f6c077848c8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:33:02.928 7f6c077848c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:33:02.929 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:33:02.929 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:33:02.929 7f6c018050f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:33:02.929 7f6c018050f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:33:02.930 7f6bfec9bcc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.930 7f6bfec9bcc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:33:02.930 7f6bfec9bcc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.931 7f6c077848c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:33:02.935 7f6c018050f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:33:02.936 7f6c046e1780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:33:02.936 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:33:02.957 7f6c077848c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:33:02.958 7f6bfe33d8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.958 7f6bfe33d8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:33:02.958 7f6bfe33d8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.967 7f6c077848c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:33:02.967 7f6bfe3e0828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.967 7f6bfe3e0828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:33:02.967 7f6bfe3e0828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.973 7f6c077848c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:33:02.982 7f6c077848c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:33:02.983 7f6bfe34f2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:02.983 7f6bfe34f2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:33:02.983 7f6bfe34f2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:03.003 7f6bfec44978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:03.003 7f6bfec44978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:33:03.003 7f6bfec44978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:03.017 7f6bfe34bf60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:03.017 7f6bfe34bf60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:33:03.017 7f6bfe34bf60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:03.017 7f6c077848c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:33:03.017 7f6c077848c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:33:03.018 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:33:03.018 7f6c077848c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1. 11/07/2024 02:33:03.023 7f6c046e1780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:33:03.030 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.'] 11/07/2024 02:33:03.047 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:33:03.053 7f6c046e1780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:33:03.059 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.'] 11/07/2024 02:33:03.531 7f6c077848c8 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:33:03.811 7f6c077848c8 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:33:04.001 7f6c077848c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue 11/07/2024 02:33:04.001 7f6c077848c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue 11/07/2024 02:33:04.061 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:04.068 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:33:04.072 7f6c046e1780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:33:04.080 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue'] 11/07/2024 02:33:04.502 7f6c077848c8 [contrail-device-manager] [WARNING]: Process 113 prepared to run in Full mode... 11/07/2024 02:33:04.503 7f6c077848c8 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:33:04.503 7f6c077848c8 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:33:04.506 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:33:04.506 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:33:04.507 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:33:04.507 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:33:04.508 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:33:04.508 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:33:04.513 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:33:04.513 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:33:04.513 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:33:04.513 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:33:04.516 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:33:04.517 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:33:04.517 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:33:04.517 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:33:04.517 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:33:04.517 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:33:04.517 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:33:04.517 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:33:04.517 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:33:04.519 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:33:04.519 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:33:04.519 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:33:04.519 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:33:04.520 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:33:04.520 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:33:04.520 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:33:04.521 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:33:04.521 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:33:04.521 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:33:04.521 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:33:04.521 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:33:04.521 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:33:04.522 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:33:04.523 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:33:04.523 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:33:04.523 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:33:04.523 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:33:04.523 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:33:04.523 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:33:04.523 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:33:04.523 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:33:04.523 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:33:04.523 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:33:04.524 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:33:04.524 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:33:04.524 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:33:04.524 7f6c077848c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:33:04.527 7f6c077848c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:33:04.573 7f6bfe083390 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:33:04.574 7f6bfe083390 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:33:04.574 7f6bfe0670f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:04.574 7f6bfe0670f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:33:04.574 7f6bfe0670f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:04.591 7f6bfe0c8198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:04.591 7f6bfe0c8198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:33:04.591 7f6bfe0c8198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:05.083 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:05.091 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:33:05.095 7f6c046e1780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:33:05.107 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:33:06.110 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:06.857 7f6bfd8c9e10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:33:06.858 7f6bfd8c9e10 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:33:06.858 7f6bfd8c9e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:33:06.858 7f6bfd8c9e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:33:06.859 7f6bfd8c9e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:33:06.859 7f6bfd8c9e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:33:06.859 7f6bfd8c9e10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:33:06.859 7f6bfd8c9e10 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:33:06.859 7f6bfd8c9e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:33:06.859 7f6bfd8c9e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:33:06.859 7f6bfd8c9e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:33:06.860 7f6bfd8c58d0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:33:06.861 7f6bfd8c58d0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:33:06.876 7f6bfd8c5cc0 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 11/07/2024 02:33:06.877 7f6bfd8c5cc0 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:33:06.877 7f6bfd8c9e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:33:06.878 7f6bfd8c9e10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:33:06.878 7f6bfd8c9e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:33:06.878 7f6bfd8c9e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:33:06.878 7f6bfd8c9e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:33:06.879 7f6bfd8c52e8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:33:06.881 7f6bfd8c52e8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:33:07.111 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:08.112 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:09.113 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:10.114 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:11.116 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:12.116 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:13.118 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:14.118 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:15.119 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:16.122 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:17.123 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:18.124 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:19.124 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:20.125 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:21.127 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:22.128 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:23.129 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:24.130 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:25.132 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:26.132 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:27.133 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:28.135 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:29.137 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:30.138 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:31.139 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:32.140 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:33.141 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:34.142 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:35.144 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:36.146 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:37.147 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:38.147 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:39.148 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:40.149 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:41.149 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:42.151 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:43.152 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:44.153 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:45.154 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:46.154 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:47.156 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:48.156 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:49.157 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:50.158 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:51.159 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:52.160 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:53.161 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:54.161 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:55.162 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:56.163 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:57.164 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:58.166 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:59.167 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:00.168 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:01.169 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:02.171 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:03.172 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:04.173 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:05.174 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:06.175 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:07.177 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:08.177 7f6c046e1780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:08.300 7f6c077848c8 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: adb9bfdc-df7a-48d0-b908-785940bb6a85 11/07/2024 02:34:08.456 7f6c077848c8 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: f1cff9ac-2ab7-4b58-8390-d193a819273b 11/07/2024 02:34:48.755 7f088f39dc08 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:34:48.755 7f088f39dc08 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:34:48.755 7f088f39dc08 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:34:48.755 7f0886a716d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:34:48.755 7f0886a716d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:34:48.756 7f0886a716d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:34:48.756 7f0886a716d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:34:48.756 7f0886a716d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.756 7f0886a716d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:34:48.756 7f0886a716d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.756 7f0886a716d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.757 7f0886a716d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 113 start_time = 1730946888756256 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:34:48.757 7f0886a716d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.795 7f088f39dc08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//'] 11/07/2024 02:34:48.800 7f088f39dc08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:34:48.801 7f08869c3240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.801 7f08869c3240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:34:48.802 7f08869c3240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.802 7f08869c3240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.802 7f08869c3240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:34:48.802 7f08869c3240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.806 7f088f39dc08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:34:48.807 7f08869c30f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.807 7f08869c30f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:34:48.808 7f08869c30f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.822 7f088f39dc08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:34:48.823 7f088691c2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.823 7f088691c2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:34:48.824 7f088691c2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.829 7f088f39dc08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:34:48.829 7f088691c2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.829 7f088691c2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:34:48.829 7f088691c2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.850 7f088f39dc08 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:34:48.851 7f088f39dc08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:34:48.851 7f088f39dc08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:34:48.851 7f088f39dc08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:34:48.851 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:34:48.852 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:34:48.852 7f08894830f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:34:48.852 7f08894830f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:34:48.853 7f088691ccc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.853 7f088691ccc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:34:48.853 7f088691ccc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.854 7f088f39dc08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:34:48.857 7f088f044630 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:34:48.857 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:34:48.861 7f08894830f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:34:48.880 7f088f39dc08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:34:48.881 7f0885fbb8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.882 7f0885fbb8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:34:48.882 7f0885fbb8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.887 7f088f39dc08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:34:48.887 7f0885fff828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.888 7f0885fff828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:34:48.888 7f0885fff828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.894 7f088f39dc08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:34:48.912 7f088f39dc08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:34:48.914 7f0885fed2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.915 7f0885fed2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:34:48.915 7f0885fed2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.937 7f08868dc390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.937 7f08868dc390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:34:48.937 7f08868dc390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.945 7f088605bac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.945 7f088605bac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:34:48.945 7f088605bac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:48.945 7f088f39dc08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:34:48.946 7f088f39dc08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:34:48.946 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:34:48.946 7f088f39dc08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1. 11/07/2024 02:34:48.954 7f088f044630 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:34:48.965 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.'] 11/07/2024 02:34:48.974 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:34:48.980 7f088f044630 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:34:48.989 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.'] 11/07/2024 02:34:49.460 7f088f39dc08 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:34:49.764 7f088f39dc08 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:34:49.993 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:49.996 7f088f39dc08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue 11/07/2024 02:34:49.997 7f088f39dc08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue 11/07/2024 02:34:50.497 7f088f39dc08 [contrail-device-manager] [WARNING]: Process 113 prepared to run in Full mode... 11/07/2024 02:34:50.498 7f088f39dc08 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:34:50.498 7f088f39dc08 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:34:50.501 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:34:50.501 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:34:50.502 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:34:50.502 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:34:50.503 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:34:50.503 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:34:50.508 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:34:50.508 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:34:50.508 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:34:50.508 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:34:50.508 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:34:50.508 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:34:50.508 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:34:50.508 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:34:50.508 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:34:50.508 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:34:50.509 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:34:50.509 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:34:50.509 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:34:50.510 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:34:50.510 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:34:50.510 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:34:50.511 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:34:50.511 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:34:50.511 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:34:50.511 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:34:50.512 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:34:50.512 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:34:50.512 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:34:50.512 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:34:50.512 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:34:50.512 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:34:50.513 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:34:50.513 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:34:50.513 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:34:50.513 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:34:50.513 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:34:50.514 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:34:50.514 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:34:50.514 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:34:50.514 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:34:50.514 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:34:50.514 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:34:50.514 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:34:50.514 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:34:50.514 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:34:50.514 7f088f39dc08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:34:50.517 7f088f39dc08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:34:50.563 7f0885e77d68 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:34:50.563 7f0885e77d68 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:34:50.564 7f0885e07198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:50.564 7f0885e07198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:34:50.564 7f0885e07198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:50.580 7f0885d5d828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:50.581 7f0885d5d828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:34:50.581 7f0885d5d828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:50.994 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:51.002 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:34:51.007 7f088f044630 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:34:51.033 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2210-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2210-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:34:52.037 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:52.756 7f0885551cc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:34:52.757 7f0885551cc0 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:34:52.757 7f0885551cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:34:52.757 7f0885551cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:34:52.757 7f0885551cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2210-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:34:52.757 7f0885551cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:34:52.757 7f0885551cc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:34:52.758 7f0885551cc0 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:34:52.758 7f0885551cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:34:52.758 7f0885551cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:34:52.758 7f0885551cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:34:52.758 7f08855576d8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:34:52.759 7f08855576d8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:34:52.765 7f0885557588 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 11/07/2024 02:34:52.765 7f0885557588 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:34:52.765 7f088761feb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:34:52.766 7f088761feb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:34:52.766 7f088761feb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:34:52.766 7f088761feb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:34:52.766 7f088761feb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:34:52.766 7f0885557048 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:34:52.768 7f0885557048 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:34:53.037 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:54.039 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:55.040 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:56.040 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:57.042 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:58.043 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:59.043 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:00.046 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:01.048 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:02.049 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:03.051 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:04.052 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:05.053 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:06.054 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:07.056 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:08.057 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:09.058 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:10.059 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:11.059 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:12.060 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:13.061 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:14.062 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:15.063 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:16.064 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:17.065 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:18.066 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:19.068 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:20.069 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:21.070 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:22.071 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:23.072 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:24.073 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:25.074 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:26.075 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:27.076 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:28.077 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:29.078 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:30.079 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:31.080 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:31.602 7f088c53a438 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:35:32.081 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:33.082 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:34.086 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:35.087 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:36.088 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:37.089 7f088f044630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process.