11/07/2024 02:23:35.973 7fd346920c48 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:23:35.974 7fd346920c48 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:23:35.974 7fd346920c48 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:23:35.974 7fd33dfa56d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:23:35.975 7fd33dfa56d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:23:35.975 7fd33dfa56d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:23:35.975 7fd33dfa56d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:23:35.975 7fd33dfa56d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:35.976 7fd33dfa56d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:23:35.976 7fd33dfa56d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:35.976 7fd33dfa56d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:35.976 7fd33dfa56d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 start_time = 1730946215975612 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:23:35.976 7fd33dfa56d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.039 7fd346920c48 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 11/07/2024 02:23:36.050 7fd346920c48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:23:36.051 7fd33df75630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.051 7fd33df75630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:23:36.051 7fd33df75630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.052 7fd33df75630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.052 7fd33df75630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:23:36.052 7fd33df75630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.055 7fd346920c48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:23:36.056 7fd33de3beb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.056 7fd33de3beb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:23:36.056 7fd33de3beb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.069 7fd346920c48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:23:36.070 7fd33de2eeb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.071 7fd33de2eeb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:23:36.071 7fd33de2eeb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.074 7fd346920c48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:23:36.075 7fd33de2eeb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.075 7fd33de2eeb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:23:36.075 7fd33de2eeb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.082 7fd346920c48 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:23:36.082 7fd346920c48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:23:36.083 7fd346920c48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:23:36.083 7fd346920c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:23:36.083 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:23:36.083 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:23:36.084 7fd33de3bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:23:36.085 7fd33de3bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:23:36.085 7fd33de2e390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.085 7fd33de2e390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:23:36.086 7fd33de2e390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.087 7fd346920c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:23:36.098 7fd343892780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:23:36.098 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:23:36.099 7fd33de3bf60 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:23:36.116 7fd346920c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:23:36.117 7fd33d4c0f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.117 7fd33d4c0f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:23:36.118 7fd33d4c0f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.123 7fd346920c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:23:36.123 7fd33dd84e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.123 7fd33dd84e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:23:36.123 7fd33dd84e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:36.128 7fd346920c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:23:36.137 7fd346920c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:23:36.310 7fd346920c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/07/2024 02:23:36.413 7fd346920c48 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 11/07/2024 02:23:36.419 7fd346920c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/07/2024 02:23:36.634 7fd346920c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/07/2024 02:23:36.707 7fd346920c48 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 11/07/2024 02:23:36.712 7fd346920c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/07/2024 02:23:36.944 7fd346920c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/07/2024 02:23:37.050 7fd346920c48 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 11/07/2024 02:23:37.054 7fd346920c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/07/2024 02:23:37.416 7fd346920c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/07/2024 02:23:38.022 7fd346920c48 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 11/07/2024 02:23:38.054 7fd346920c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 11/07/2024 02:23:38.054 7fd33d4c0f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:38.059 7fd33d4c0f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:23:38.059 7fd33d4c0f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:38.072 7fd346920c48 [contrail-device-manager] [ERROR]: Error in db walk read 'type' 11/07/2024 02:23:38.073 7fd33d4cba20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:38.075 7fd33d4cba20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:23:38.075 7fd33d4cba20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:38.097 7fd33d4cd8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:38.098 7fd33d4cd8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:23:38.098 7fd33d4cd8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:38.098 7fd346920c48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:23:38.098 7fd346920c48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:23:38.098 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:23:38.099 7fd346920c48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1. 11/07/2024 02:23:38.102 7fd343892780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:23:38.111 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:23:38.209 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:23:38.215 7fd343892780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:23:38.221 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:23:38.610 7fd346920c48 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:23:38.765 7fd346920c48 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:23:38.979 7fd346920c48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue 11/07/2024 02:23:38.979 7fd346920c48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue 11/07/2024 02:23:39.223 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:39.228 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:23:39.231 7fd343892780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:23:39.244 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue'] 11/07/2024 02:23:39.480 7fd346920c48 [contrail-device-manager] [WARNING]: Process 114 prepared to run in Full mode... 11/07/2024 02:23:39.480 7fd346920c48 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:23:39.480 7fd346920c48 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:23:39.483 7fd346920c48 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:23:39.483 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:23:39.484 7fd346920c48 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:23:39.484 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:23:39.485 7fd346920c48 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:23:39.485 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:23:39.489 7fd346920c48 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:23:39.489 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:23:39.489 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:23:39.489 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:23:39.489 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:23:39.489 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:23:39.489 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:23:39.489 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:23:39.490 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:23:39.490 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:23:39.490 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:23:39.490 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:23:39.490 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:23:39.491 7fd346920c48 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:23:39.491 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:23:39.491 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:23:39.492 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:23:39.492 7fd346920c48 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:23:39.492 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:23:39.492 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:23:39.493 7fd346920c48 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:23:39.493 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:23:39.493 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:23:39.493 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:23:39.493 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:23:39.493 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:23:39.494 7fd346920c48 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:23:39.494 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:23:39.494 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:23:39.494 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:23:39.494 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:23:39.494 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:23:39.494 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:23:39.495 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:23:39.495 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:23:39.495 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:23:39.495 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:23:39.495 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:23:39.495 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:23:39.495 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:23:39.496 7fd346920c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:23:39.499 7fd346920c48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:23:39.558 7fd33d3a1438 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:23:39.558 7fd33d3a1438 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:23:39.559 7fd33d3dad68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:39.559 7fd33d3dad68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:23:39.559 7fd33d3dad68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:39.564 7fd33d3e38d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:39.564 7fd33d3e38d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:23:39.564 7fd33d3e38d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:39.975 7fd33cb25f60 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:23:39.976 7fd33cb25f60 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:23:39.976 7fd33cb25f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:23:39.976 7fd33cb25f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:23:39.976 7fd33cb25f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:23:39.976 7fd33cb25f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:23:39.976 7fd33cb25f60 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:23:39.977 7fd33cb25f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:23:39.977 7fd33cb25f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:39.977 7fd33cb25f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:23:39.977 7fd33cb25f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:39.977 7fd33cb25f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:39.977 7fd33cb25f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:23:39.977 7fd33cb25f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:40.247 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:40.256 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:23:40.260 7fd343892780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:23:40.271 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:23:41.274 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:42.275 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:42.666 7fd33cb19c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:42.666 7fd33cb19c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:23:42.666 7fd33cb19c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:43.276 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:43.977 7fd33ca48780 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:23:43.978 7fd33ca48780 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:23:43.979 7fd33ca48780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:23:43.979 7fd33ca48780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:23:43.979 7fd33ca48780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:23:43.979 7fd33ca48780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:23:43.979 7fd33ca48780 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:23:43.979 7fd33ca48780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:23:43.979 7fd33ca48780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:43.979 7fd33ca48780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:23:43.980 7fd33ca48780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:43.980 7fd33ca48780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:43.980 7fd33ca48780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:23:43.980 7fd33ca48780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:44.277 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:45.278 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:46.279 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:47.280 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:47.979 7fd33cabfac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:23:47.980 7fd33cabfac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:23:47.980 7fd33cabfac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:23:47.980 7fd33cabfac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:23:47.980 7fd33cabfac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:23:47.980 7fd33cabfac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:23:47.980 7fd33cabfac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:23:47.981 7fd33cabfac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:23:47.981 7fd33cabfac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:47.981 7fd33cabfac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:23:47.981 7fd33cabfac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:47.981 7fd33cabfac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:47.981 7fd33cabfac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:23:47.981 7fd33cabfac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:48.281 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:49.282 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:50.283 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:51.284 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:51.980 7fd33ca3bb70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:23:51.981 7fd33ca3bb70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:23:51.981 7fd33ca3bb70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:23:51.981 7fd33ca3bb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:23:51.981 7fd33ca3bb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:23:51.982 7fd33ca3bb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:23:51.982 7fd33ca3bb70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:23:51.982 7fd33ca3bb70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:23:51.982 7fd33ca3bb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:51.982 7fd33ca3bb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:23:51.982 7fd33ca3bb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:51.982 7fd33ca3bb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:51.982 7fd33ca3bb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:23:51.983 7fd33ca3bb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:52.285 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:53.285 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:54.287 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:55.287 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:55.982 7fd33ca3bc18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:23:55.983 7fd33ca3bc18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:23:55.983 7fd33ca3bc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:23:55.983 7fd33ca3bc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:23:55.983 7fd33ca3bc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:23:55.983 7fd33ca3bc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:23:55.983 7fd33ca3bc18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:23:55.983 7fd33ca3bc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:23:55.983 7fd33ca3bc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:55.984 7fd33ca3bc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:23:55.984 7fd33ca3bc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:55.984 7fd33ca3bc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:55.984 7fd33ca3bc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:23:55.984 7fd33ca3bc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:56.289 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:57.289 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:58.290 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:59.291 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:59.983 7fd33cb19240 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:23:59.985 7fd33cb19240 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:23:59.985 7fd33cb19240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:23:59.985 7fd33cb19240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:23:59.985 7fd33cb19240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:23:59.985 7fd33cb19240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:23:59.985 7fd33cb19240 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:23:59.986 7fd33cb19240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:23:59.986 7fd33cb19240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:59.986 7fd33cb19240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:23:59.986 7fd33cb19240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:59.986 7fd33cb19240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:59.986 7fd33cb19240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:23:59.986 7fd33cb19240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:00.292 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:01.293 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:02.294 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:03.295 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:03.986 7fd343a48c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:24:03.987 7fd343a48c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:24:03.987 7fd343a48c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:24:03.987 7fd343a48c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:03.988 7fd343a48c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:24:03.988 7fd343a48c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:03.988 7fd343a48c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:24:03.988 7fd343a48c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:24:03.988 7fd343a48c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:03.988 7fd343a48c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:24:03.988 7fd343a48c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:03.988 7fd343a48c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:03.988 7fd343a48c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:24:03.988 7fd343a48c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:04.295 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:05.296 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:06.297 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:07.298 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:07.988 7fd343a0a438 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:24:07.989 7fd343a0a438 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:24:07.989 7fd343a0a438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:24:07.989 7fd343a0a438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:07.990 7fd343a0a438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:24:07.990 7fd343a0a438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:07.990 7fd343a0a438 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:24:07.990 7fd343a0a438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:24:07.990 7fd343a0a438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:07.990 7fd343a0a438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:24:07.990 7fd343a0a438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:07.990 7fd343a0a438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:07.991 7fd343a0a438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:24:07.991 7fd343a0a438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:08.299 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:09.300 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:10.301 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:11.302 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:11.990 7fd3439fbc18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:24:11.991 7fd3439fbc18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:24:11.991 7fd3439fbc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:24:11.991 7fd3439fbc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:11.991 7fd3439fbc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:24:11.991 7fd3439fbc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:11.991 7fd3439fbc18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:24:11.992 7fd3439fbc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:24:11.992 7fd3439fbc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:11.992 7fd3439fbc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:24:11.992 7fd3439fbc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:11.992 7fd3439fbc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:11.992 7fd3439fbc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:24:11.992 7fd3439fbc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:12.303 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:13.304 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:14.305 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:15.305 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:15.992 7fd33c9d64e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:24:15.993 7fd33c9d64e0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:24:15.993 7fd33c9d64e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:24:15.993 7fd33c9d64e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:15.994 7fd33c9d64e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:24:15.994 7fd33c9d64e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:15.994 7fd33c9d64e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:24:15.994 7fd33c9d64e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:24:15.994 7fd33c9d64e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:15.994 7fd33c9d64e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:24:15.994 7fd33c9d64e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:15.994 7fd33c9d64e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:15.995 7fd33c9d64e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:24:15.995 7fd33c9d64e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:16.306 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:17.307 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:18.308 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:19.309 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:19.994 7fd33cabfac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:24:19.995 7fd33cabfac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:24:19.995 7fd33cabfac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:24:19.995 7fd33cabfac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:19.995 7fd33cabfac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:24:19.995 7fd33cabfac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:19.995 7fd33cabfac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:24:19.996 7fd33cabfac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:24:19.996 7fd33cabfac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:19.996 7fd33cabfac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:24:19.996 7fd33cabfac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:19.996 7fd33cabfac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:19.996 7fd33cabfac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:24:19.996 7fd33cabfac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:20.310 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:21.311 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:22.311 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:23.312 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:23.996 7fd33c9caac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:24:23.996 7fd33c9caac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:24:23.997 7fd33c9caac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:24:23.997 7fd33c9caac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:23.997 7fd33c9caac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:24:23.997 7fd33c9caac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:23.997 7fd33c9caac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:24:23.997 7fd33c9caac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:24:23.997 7fd33c9caac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:23.997 7fd33c9caac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:24:23.997 7fd33c9caac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:23.998 7fd33c9caac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:23.998 7fd33c9caac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:24:23.998 7fd33c9caac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:24.313 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:25.313 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:26.315 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:27.316 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:27.997 7fd33c9cab70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:24:27.998 7fd33c9cab70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:24:27.998 7fd33c9cab70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:24:27.998 7fd33c9cab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:27.998 7fd33c9cab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:24:27.998 7fd33c9cab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:27.998 7fd33c9cab70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:24:27.999 7fd33c9cab70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:24:27.999 7fd33c9cab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:27.999 7fd33c9cab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:24:27.999 7fd33c9cab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:27.999 7fd33c9cab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:27.999 7fd33c9cab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:24:27.999 7fd33c9cab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:28.317 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:29.318 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:30.320 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:31.321 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:31.999 7fd33cb2a438 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:24:32.002 7fd33cb2a438 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:24:32.002 7fd33cb2a438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:24:32.002 7fd33cb2a438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:32.002 7fd33cb2a438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:24:32.002 7fd33cb2a438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:32.003 7fd33cb2a438 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:24:32.003 7fd33cb2a438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:24:32.003 7fd33cb2a438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.003 7fd33cb2a438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:24:32.003 7fd33cb2a438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.003 7fd33cb2a438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.003 7fd33cb2a438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:24:32.003 7fd33cb2a438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.322 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:33.322 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:34.324 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:35.325 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:36.003 7fd33cb2a4e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:24:36.004 7fd33cb2a4e0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:24:36.004 7fd33cb2a4e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:24:36.004 7fd33cb2a4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:36.004 7fd33cb2a4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:24:36.004 7fd33cb2a4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:36.004 7fd33cb2a4e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:24:36.005 7fd33cb2a4e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:24:36.005 7fd33cb2a4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:36.005 7fd33cb2a4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:24:36.005 7fd33cb2a4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:36.005 7fd33cb2a4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:36.005 7fd33cb2a4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:24:36.005 7fd33cb2a4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:36.326 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:37.327 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:38.328 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:39.329 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:40.005 7fd33ca3bac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:24:40.005 7fd33ca3bac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:24:40.006 7fd33ca3bac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:24:40.006 7fd33ca3bac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:40.006 7fd33ca3bac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:24:40.006 7fd33ca3bac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:40.006 7fd33ca3bac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:24:40.006 7fd33ca3bac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:24:40.006 7fd33ca3bac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:40.006 7fd33ca3bac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:24:40.007 7fd33ca3bac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:40.007 7fd33ca3bac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:40.007 7fd33ca3bac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:24:40.007 7fd33ca3bac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:40.330 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:41.331 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:42.332 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:43.332 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:44.006 7fd33cb20780 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:24:44.007 7fd33cb20780 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:24:44.007 7fd33cb20780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:24:44.007 7fd33cb20780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:44.008 7fd33cb20780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:24:44.008 7fd33cb20780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:44.008 7fd33cb20780 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:24:44.008 7fd33cb20780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:24:44.008 7fd33cb20780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:44.008 7fd33cb20780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:24:44.008 7fd33cb20780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:44.008 7fd33cb20780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:44.008 7fd33cb20780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946215975612 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:24:44.009 7fd33cb20780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:44.333 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:45.334 7fd343892780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:35.642 7f38bc44ed48 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:25:35.642 7f38bc44ed48 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:25:35.642 7f38bc44ed48 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:25:35.643 7f38b3b176d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:25:35.643 7f38b3b176d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:25:35.643 7f38b3b176d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:25:35.643 7f38b3b176d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:25:35.643 7f38b3b176d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.644 7f38b3b176d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:25:35.644 7f38b3b176d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.644 7f38b3b176d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.644 7f38b3b176d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 start_time = 1730946335643644 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:25:35.644 7f38b3b176d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.677 7f38bc44ed48 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 11/07/2024 02:25:35.682 7f38bc44ed48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:25:35.682 7f38b3a66240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.683 7f38b3a66240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:25:35.683 7f38b3a66240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.683 7f38b3a66240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.683 7f38b3a66240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:25:35.683 7f38b3a66240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.690 7f38bc44ed48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:25:35.690 7f38b3a660f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.690 7f38b3a660f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:25:35.690 7f38b3a660f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.697 7f38bc44ed48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:25:35.697 7f38b39bf2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.697 7f38b39bf2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:25:35.697 7f38b39bf2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.708 7f38bc44ed48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:25:35.708 7f38b39bf240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.708 7f38b39bf240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:25:35.709 7f38b39bf240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.721 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:25:35.721 7f38bc44ed48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:25:35.722 7f38bc44ed48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:25:35.722 7f38bc44ed48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:25:35.722 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:25:35.722 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:25:35.723 7f38b652a0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:25:35.723 7f38b652a0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:25:35.723 7f38b39bfcc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.723 7f38b39bfcc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:25:35.723 7f38b39bfcc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.724 7f38bc44ed48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:25:35.727 7f38b9406780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:25:35.728 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:25:35.728 7f38b652a0f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:25:35.762 7f38bc44ed48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:25:35.763 7f38b3861780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.763 7f38b3861780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:25:35.763 7f38b3861780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.768 7f38bc44ed48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:25:35.768 7f38b3866c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.768 7f38b3866c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:25:35.768 7f38b3866c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.772 7f38bc44ed48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:25:35.780 7f38bc44ed48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:25:35.781 7f38b38756d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.781 7f38b38756d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:25:35.781 7f38b38756d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.795 7f38b3980390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.795 7f38b3980390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:25:35.795 7f38b3980390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.801 7f38b386eac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.801 7f38b386eac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:25:35.801 7f38b386eac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:35.801 7f38bc44ed48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:25:35.801 7f38bc44ed48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:25:35.802 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:25:35.802 7f38bc44ed48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1. 11/07/2024 02:25:35.805 7f38b9406780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:25:35.812 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:25:35.820 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:25:35.824 7f38b9406780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:25:35.830 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:25:36.312 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:25:36.494 7f38bc44ed48 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:25:36.662 7f38bc44ed48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue 11/07/2024 02:25:36.663 7f38bc44ed48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue 11/07/2024 02:25:36.832 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:36.839 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:25:36.842 7f38b9406780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:25:36.852 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue'] 11/07/2024 02:25:37.164 7f38bc44ed48 [contrail-device-manager] [WARNING]: Process 114 prepared to run in Full mode... 11/07/2024 02:25:37.164 7f38bc44ed48 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:25:37.164 7f38bc44ed48 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:25:37.167 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:25:37.167 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:25:37.168 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:25:37.168 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:25:37.169 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:25:37.169 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:25:37.173 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:25:37.173 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:25:37.173 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:25:37.174 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:25:37.174 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:25:37.174 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:25:37.174 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:25:37.174 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:25:37.174 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:25:37.174 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:25:37.174 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:25:37.174 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:25:37.174 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:25:37.176 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:25:37.176 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:25:37.176 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:25:37.176 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:25:37.176 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:25:37.176 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:25:37.177 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:25:37.177 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:25:37.177 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:25:37.177 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:25:37.177 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:25:37.177 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:25:37.177 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:25:37.178 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:25:37.178 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:25:37.178 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:25:37.178 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:25:37.179 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:25:37.179 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:25:37.179 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:25:37.179 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:25:37.179 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:25:37.179 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:25:37.179 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:25:37.179 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:25:37.179 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:25:37.179 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:25:37.180 7f38bc44ed48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:25:37.182 7f38bc44ed48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:25:37.222 7f38b3656a20 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:25:37.223 7f38b3656a20 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:25:37.223 7f38b38126d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:37.223 7f38b38126d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:25:37.223 7f38b38126d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:37.237 7f38b37af978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:37.237 7f38b37af978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:25:37.237 7f38b37af978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:37.854 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:37.863 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:25:37.866 7f38b9406780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:25:37.878 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:25:38.880 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:39.643 7f38b2de3048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:25:39.644 7f38b2de3048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:25:39.644 7f38b2de3048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:25:39.644 7f38b2de3048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:39.645 7f38b2de3048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:25:39.645 7f38b2de3048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:39.645 7f38b2de3048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:25:39.645 7f38b2de3048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:25:39.645 7f38b2de3048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:39.645 7f38b2de3048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:25:39.645 7f38b2de3048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:39.645 7f38b2de3048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:39.646 7f38b2de3048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:25:39.646 7f38b2de3048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:39.881 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:40.882 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:41.883 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:42.884 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:43.645 7f38b2ddca20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:25:43.646 7f38b2ddca20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:25:43.646 7f38b2ddca20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:25:43.646 7f38b2ddca20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:43.646 7f38b2ddca20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:25:43.647 7f38b2ddca20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:43.647 7f38b2ddca20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:25:43.647 7f38b2ddca20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:25:43.647 7f38b2ddca20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:43.647 7f38b2ddca20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:25:43.647 7f38b2ddca20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:43.647 7f38b2ddca20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:43.647 7f38b2ddca20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:25:43.647 7f38b2ddca20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:43.885 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:44.886 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:45.887 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:46.888 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:47.647 7f38b2ddcf60 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:25:47.648 7f38b2ddcf60 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:25:47.648 7f38b2ddcf60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:25:47.648 7f38b2ddcf60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:47.648 7f38b2ddcf60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:25:47.649 7f38b2ddcf60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:47.649 7f38b2ddcf60 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:25:47.649 7f38b2ddcf60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:25:47.649 7f38b2ddcf60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:47.649 7f38b2ddcf60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:25:47.649 7f38b2ddcf60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:47.649 7f38b2ddcf60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:47.649 7f38b2ddcf60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:25:47.650 7f38b2ddcf60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:47.924 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:48.925 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:49.927 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:50.928 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:51.649 7f38b396fac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:25:51.650 7f38b396fac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:25:51.650 7f38b396fac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:25:51.650 7f38b396fac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:51.650 7f38b396fac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:25:51.650 7f38b396fac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:51.650 7f38b396fac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:25:51.651 7f38b396fac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:25:51.651 7f38b396fac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:51.651 7f38b396fac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:25:51.651 7f38b396fac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:51.651 7f38b396fac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:51.651 7f38b396fac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:25:51.651 7f38b396fac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:51.929 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:52.930 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:53.931 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:54.931 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:55.650 7f38b2de3f60 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:25:55.651 7f38b2de3f60 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:25:55.651 7f38b2de3f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:25:55.651 7f38b2de3f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:55.652 7f38b2de3f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:25:55.652 7f38b2de3f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:55.652 7f38b2de3f60 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:25:55.652 7f38b2de3f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:25:55.652 7f38b2de3f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:55.652 7f38b2de3f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:25:55.652 7f38b2de3f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:55.652 7f38b2de3f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:55.652 7f38b2de3f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:25:55.652 7f38b2de3f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:55.932 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:56.934 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:57.935 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:58.936 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:59.652 7f38b95bce10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:25:59.653 7f38b95bce10 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:25:59.653 7f38b95bce10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:25:59.653 7f38b95bce10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:59.653 7f38b95bce10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:25:59.653 7f38b95bce10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:25:59.653 7f38b95bce10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:25:59.654 7f38b95bce10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:25:59.654 7f38b95bce10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:59.654 7f38b95bce10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:25:59.654 7f38b95bce10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:59.654 7f38b95bce10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:59.654 7f38b95bce10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:25:59.654 7f38b95bce10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:25:59.937 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:00.938 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:01.939 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:02.941 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:03.653 7f38b95bce10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:03.654 7f38b95bce10 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:03.655 7f38b95bce10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:03.655 7f38b95bce10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:03.655 7f38b95bce10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:03.655 7f38b95bce10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:03.655 7f38b95bce10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:03.655 7f38b95bce10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:03.655 7f38b95bce10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:03.655 7f38b95bce10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:03.655 7f38b95bce10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:03.656 7f38b95bce10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:03.656 7f38b95bce10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:03.656 7f38b95bce10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:03.942 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:04.943 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:05.943 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:06.944 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:07.655 7f38b2dfbf60 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:07.658 7f38b2dfbf60 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:07.658 7f38b2dfbf60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:07.658 7f38b2dfbf60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:07.658 7f38b2dfbf60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:07.658 7f38b2dfbf60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:07.658 7f38b2dfbf60 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:07.659 7f38b2dfbf60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:07.659 7f38b2dfbf60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:07.659 7f38b2dfbf60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:07.659 7f38b2dfbf60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:07.659 7f38b2dfbf60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:07.659 7f38b2dfbf60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:07.659 7f38b2dfbf60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:07.945 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:08.946 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:09.947 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:10.948 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:11.659 7f38b2e0a438 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:11.660 7f38b2e0a438 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:11.660 7f38b2e0a438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:11.660 7f38b2e0a438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:11.660 7f38b2e0a438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:11.660 7f38b2e0a438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:11.660 7f38b2e0a438 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:11.660 7f38b2e0a438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:11.661 7f38b2e0a438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:11.661 7f38b2e0a438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:11.661 7f38b2e0a438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:11.661 7f38b2e0a438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:11.661 7f38b2e0a438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:11.661 7f38b2e0a438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:11.949 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:12.950 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:13.951 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:14.952 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:15.661 7f38b3866ac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:15.662 7f38b3866ac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:15.662 7f38b3866ac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:15.662 7f38b3866ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:15.662 7f38b3866ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:15.662 7f38b3866ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:15.662 7f38b3866ac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:15.663 7f38b3866ac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:15.663 7f38b3866ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:15.663 7f38b3866ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:15.663 7f38b3866ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:15.663 7f38b3866ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:15.663 7f38b3866ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:15.663 7f38b3866ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:15.953 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:16.954 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:17.955 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:18.956 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:19.663 7f38b2e0d438 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:19.664 7f38b2e0d438 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:19.664 7f38b2e0d438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:19.664 7f38b2e0d438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:19.664 7f38b2e0d438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:19.664 7f38b2e0d438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:19.664 7f38b2e0d438 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:19.665 7f38b2e0d438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:19.665 7f38b2e0d438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:19.665 7f38b2e0d438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:19.665 7f38b2e0d438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:19.665 7f38b2e0d438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:19.665 7f38b2e0d438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:19.665 7f38b2e0d438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:19.957 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:20.959 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:21.960 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:22.961 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:23.664 7f38b2e0dac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:23.665 7f38b2e0dac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:23.665 7f38b2e0dac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:23.665 7f38b2e0dac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:23.665 7f38b2e0dac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:23.665 7f38b2e0dac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:23.666 7f38b2e0dac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:23.666 7f38b2e0dac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:23.666 7f38b2e0dac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:23.666 7f38b2e0dac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:23.666 7f38b2e0dac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:23.666 7f38b2e0dac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:23.666 7f38b2e0dac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:23.666 7f38b2e0dac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:23.962 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:24.963 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:25.964 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:26.964 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:27.666 7f38b2e10438 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:27.667 7f38b2e10438 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:27.667 7f38b2e10438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:27.667 7f38b2e10438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:27.667 7f38b2e10438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:27.667 7f38b2e10438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:27.667 7f38b2e10438 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:27.668 7f38b2e10438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:27.668 7f38b2e10438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:27.668 7f38b2e10438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:27.668 7f38b2e10438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:27.668 7f38b2e10438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:27.668 7f38b2e10438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:27.668 7f38b2e10438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:27.965 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:28.967 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:29.968 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:30.969 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:31.667 7f38b95d3828 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:31.669 7f38b95d3828 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:31.669 7f38b95d3828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:31.669 7f38b95d3828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:31.670 7f38b95d3828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:31.670 7f38b95d3828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:31.670 7f38b95d3828 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:31.670 7f38b95d3828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:31.671 7f38b95d3828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:31.671 7f38b95d3828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:31.671 7f38b95d3828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:31.671 7f38b95d3828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:31.671 7f38b95d3828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:31.671 7f38b95d3828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:31.970 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:32.970 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:33.971 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:34.973 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:35.669 7f38b2e10630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:35.670 7f38b2e10630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:26:35.670 7f38b2e10630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:35.671 7f38b2e10630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:35.671 7f38b2e10630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:35.671 7f38b2e10630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:35.671 7f38b2e10630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:26:35.671 7f38b2e10630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:26:35.671 7f38b2e10630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:35.671 7f38b2e10630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/07/2024 02:26:35.672 7f38b2e10630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:35.672 7f38b2e10630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:35.672 7f38b2e10630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 114 http_port = 8096 start_time = 1730946335643644 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:26:35.672 7f38b2e10630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:35.973 7f38b9406780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:36.307 7f38bc44ed48 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 5ba92628-0e8f-46ff-9c5b-baf2735173be 11/07/2024 02:26:36.459 7f38bc44ed48 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 155e72c1-f8e6-4c46-b897-708544212d8a 11/07/2024 02:27:21.754 7fb20ced9708 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:27:21.754 7fb20ced9708 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:27:21.754 7fb20ced9708 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:27:21.754 7fb2045726d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:27:21.754 7fb2045726d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:27:21.755 7fb2045726d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:27:21.755 7fb2045726d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:27:21.755 7fb2045726d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.755 7fb2045726d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:21.755 7fb2045726d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.755 7fb2045726d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.755 7fb2045726d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 113 start_time = 1730946441754986 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:27:21.756 7fb2045726d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.790 7fb20ced9708 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 11/07/2024 02:27:21.794 7fb20ced9708 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:27:21.795 7fb2044c3240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.795 7fb2044c3240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:27:21.795 7fb2044c3240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.795 7fb2044c3240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.796 7fb2044c3240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:27:21.796 7fb2044c3240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.799 7fb20ced9708 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:27:21.799 7fb2044c30f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.799 7fb2044c30f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:21.799 7fb2044c30f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.806 7fb20ced9708 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:27:21.807 7fb20441b2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.807 7fb20441b2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:27:21.807 7fb20441b2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.809 7fb20ced9708 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:27:21.809 7fb20441b240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.809 7fb20441b240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:21.809 7fb20441b240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.816 7fb20ced9708 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:27:21.816 7fb20ced9708 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:27:21.816 7fb20ced9708 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:27:21.816 7fb20ced9708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:27:21.817 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:27:21.817 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:21.817 7fb206f870f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:27:21.817 7fb206f870f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:27:21.817 7fb20441bcc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.818 7fb20441bcc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:27:21.818 7fb20441bcc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.819 7fb20ced9708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:27:21.823 7fb209e62780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:21.823 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:27:21.824 7fb206f870f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:27:21.842 7fb20ced9708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:27:21.843 7fb2042be780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.844 7fb2042be780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:27:21.844 7fb2042be780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.848 7fb20ced9708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:27:21.848 7fb2042c2c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.848 7fb2042c2c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:27:21.848 7fb2042c2c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.855 7fb20ced9708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:27:21.863 7fb20ced9708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:27:21.864 7fb2042d06d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.864 7fb2042d06d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:21.864 7fb2042d06d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.875 7fb208ac4a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.876 7fb208ac4a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:27:21.876 7fb208ac4a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.881 7fb2043accc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.882 7fb2043accc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:21.882 7fb2043accc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.882 7fb20ced9708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:27:21.882 7fb20ced9708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:27:21.882 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:21.883 7fb20ced9708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1. 11/07/2024 02:27:21.886 7fb209e62780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:21.892 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:27:21.899 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:21.903 7fb209e62780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:21.909 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:27:22.395 7fb20ced9708 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:27:22.660 7fb20ced9708 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:27:22.849 7fb20ced9708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue 11/07/2024 02:27:22.849 7fb20ced9708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue 11/07/2024 02:27:22.911 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:22.918 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:22.921 7fb209e62780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:22.929 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue'] 11/07/2024 02:27:23.350 7fb20ced9708 [contrail-device-manager] [WARNING]: Process 113 prepared to run in Full mode... 11/07/2024 02:27:23.351 7fb20ced9708 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:27:23.351 7fb20ced9708 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:27:23.354 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:27:23.354 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:27:23.355 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:27:23.355 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:27:23.356 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:27:23.356 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:27:23.360 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:27:23.360 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:27:23.360 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:27:23.360 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:27:23.360 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:27:23.361 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:27:23.361 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:27:23.361 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:27:23.361 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:27:23.361 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:27:23.361 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:27:23.361 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:27:23.361 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:27:23.363 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:27:23.363 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:27:23.363 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:27:23.363 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:27:23.364 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:27:23.364 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:27:23.364 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:27:23.364 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:27:23.364 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:27:23.364 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:27:23.364 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:27:23.365 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:27:23.365 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:27:23.366 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:27:23.366 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:27:23.366 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:27:23.366 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:27:23.366 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:27:23.366 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:27:23.366 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:27:23.366 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:27:23.366 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:27:23.366 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:27:23.366 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:27:23.367 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:27:23.367 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:27:23.367 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:27:23.367 7fb20ced9708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:27:23.370 7fb20ced9708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:27:23.414 7fb2041d8b70 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:27:23.414 7fb2041d8b70 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:27:23.414 7fb20409f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:23.414 7fb20409f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:27:23.414 7fb20409f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:23.429 7fb1fff41390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:23.429 7fb1fff41390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:23.430 7fb1fff41390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:23.932 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:23.943 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:23.947 7fb209e62780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:23.968 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:27:24.973 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:25.755 7fb1ff81b6d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:27:25.756 7fb1ff81b6d8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:27:25.756 7fb1ff81b6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:27:25.756 7fb1ff81b6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:27:25.757 7fb1ff81b6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:27:25.757 7fb1ff81b6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:27:25.757 7fb1ff81b6d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:27:25.757 7fb1ff81b6d8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:27:25.757 7fb1ff81b6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:27:25.758 7fb1ff81b6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:27:25.758 7fb1ff81b6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:27:25.758 7fb1ff82d198 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:27:25.759 7fb1ff82d198 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:27:25.763 7fb1ff82d048 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 11/07/2024 02:27:25.764 7fb1ff82d048 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:27:25.764 7fb205120eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:27:25.765 7fb205120eb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:27:25.765 7fb205120eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:27:25.765 7fb205120eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:27:25.765 7fb205120eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:27:25.766 7fb1ff814eb8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:27:25.767 7fb1ff814eb8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:27:25.974 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:26.975 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:27.976 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:28.976 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:29.978 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:30.978 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:31.979 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:32.980 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:33.980 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:34.981 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:35.981 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:36.982 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:37.983 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:38.984 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:39.985 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:40.986 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:41.987 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:42.987 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:43.988 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:44.989 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:45.990 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:46.992 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:47.993 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:48.994 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:49.995 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:50.996 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:51.997 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:52.998 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:53.999 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:55.000 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:56.001 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:57.002 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:58.002 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:59.003 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:00.005 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:01.006 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:02.007 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:03.008 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:04.009 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:05.010 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:06.012 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:07.013 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:08.014 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:09.015 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:10.015 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:11.016 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:12.017 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:13.017 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:14.018 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:15.020 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:16.021 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:17.022 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:18.023 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:19.023 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:20.024 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:21.025 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:22.026 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:22.935 7fb20ced9708 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 5ba92628-0e8f-46ff-9c5b-baf2735173be 11/07/2024 02:28:23.027 7fb209e62780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:23.095 7fb20ced9708 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 155e72c1-f8e6-4c46-b897-708544212d8a 11/07/2024 02:29:08.371 7fdf79f5cd08 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:29:08.372 7fdf79f5cd08 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:29:08.372 7fdf79f5cd08 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:29:08.372 7fdf715cb6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:29:08.372 7fdf715cb6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:29:08.373 7fdf715cb6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:29:08.373 7fdf715cb6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:29:08.373 7fdf715cb6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.373 7fdf715cb6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:08.373 7fdf715cb6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.373 7fdf715cb6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.373 7fdf715cb6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 113 start_time = 1730946548373064 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:29:08.373 7fdf715cb6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.424 7fdf79f5cd08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 11/07/2024 02:29:08.433 7fdf79f5cd08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:29:08.433 7fdf7151b240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.433 7fdf7151b240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:29:08.433 7fdf7151b240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.433 7fdf7151b240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.433 7fdf7151b240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:29:08.434 7fdf7151b240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.437 7fdf79f5cd08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:29:08.437 7fdf7151b240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.438 7fdf7151b240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:08.438 7fdf7151b240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.445 7fdf79f5cd08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:29:08.446 7fdf714752e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.446 7fdf714752e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:29:08.446 7fdf714752e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.449 7fdf79f5cd08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:29:08.449 7fdf714752e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.450 7fdf714752e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:08.450 7fdf714752e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.460 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:29:08.461 7fdf79f5cd08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:29:08.461 7fdf79f5cd08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:29:08.461 7fdf79f5cd08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:29:08.461 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:29:08.461 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:08.462 7fdf73fdf0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:29:08.462 7fdf73fdf0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:29:08.462 7fdf71475cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.462 7fdf71475cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:29:08.462 7fdf71475cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.463 7fdf79f5cd08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:29:08.466 7fdf73fdf0f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:29:08.467 7fdf7144ee10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:08.467 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:29:08.485 7fdf79f5cd08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:29:08.486 7fdf71317780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.487 7fdf71317780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:29:08.487 7fdf71317780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.492 7fdf79f5cd08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:29:08.492 7fdf7131ac18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.492 7fdf7131ac18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:29:08.492 7fdf7131ac18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.496 7fdf79f5cd08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:29:08.503 7fdf79f5cd08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:29:08.503 7fdf713286d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.503 7fdf713286d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:08.503 7fdf713286d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.522 7fdf714340f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.523 7fdf714340f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:29:08.523 7fdf714340f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.531 7fdf71405d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.531 7fdf71405d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:08.531 7fdf71405d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:08.531 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:29:08.531 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:29:08.531 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:08.532 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1. 11/07/2024 02:29:08.535 7fdf7144ee10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:08.541 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:29:08.548 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:08.552 7fdf7144ee10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:08.557 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:29:09.048 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:29:09.307 7fdf79f5cd08 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:29:09.525 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue 11/07/2024 02:29:09.525 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue 11/07/2024 02:29:09.559 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:09.565 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:09.568 7fdf7144ee10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:09.576 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue'] 11/07/2024 02:29:10.026 7fdf79f5cd08 [contrail-device-manager] [WARNING]: Process 113 prepared to run in Full mode... 11/07/2024 02:29:10.027 7fdf79f5cd08 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:29:10.027 7fdf79f5cd08 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:29:10.032 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:29:10.032 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:29:10.034 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:29:10.034 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:29:10.036 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:29:10.036 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:29:10.044 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:29:10.044 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:29:10.044 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:29:10.045 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:29:10.045 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:29:10.045 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:29:10.045 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:29:10.045 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:29:10.045 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:29:10.045 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:29:10.045 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:29:10.045 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:29:10.045 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:29:10.047 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:29:10.047 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:29:10.047 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:29:10.047 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:29:10.048 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:29:10.048 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:29:10.048 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:29:10.048 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:29:10.048 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:29:10.048 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:29:10.048 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:29:10.048 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:29:10.049 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:29:10.049 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:29:10.049 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:29:10.050 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:29:10.050 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:29:10.050 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:29:10.050 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:29:10.050 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:29:10.050 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:29:10.050 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:29:10.050 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:29:10.050 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:29:10.051 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:29:10.051 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:29:10.051 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:29:10.051 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:29:10.053 7fdf79f5cd08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:29:10.141 7fdf710600f0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:29:10.141 7fdf710600f0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:29:10.141 7fdf70fef390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:10.141 7fdf70fef390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:29:10.142 7fdf70fef390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:10.172 7fdf70fe0cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:10.173 7fdf70fe0cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:10.173 7fdf70fe0cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:10.579 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:10.588 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:10.592 7fdf7144ee10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:10.604 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:29:10.669 7fdf7094c780 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:29:11.608 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:12.372 7fdf7089c198 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:29:12.373 7fdf7089c198 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:29:12.374 7fdf7089c198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:29:12.374 7fdf7089c198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:29:12.374 7fdf7089c198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:29:12.374 7fdf7089c198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:29:12.374 7fdf7089c198 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:29:12.374 7fdf7089c198 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:29:12.375 7fdf7089c198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:29:12.375 7fdf7089c198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:29:12.375 7fdf7089c198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:29:12.375 7fdf7089c4e0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:29:12.376 7fdf7089c4e0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:29:12.378 7fdf7089c048 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 11/07/2024 02:29:12.379 7fdf7089c048 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:29:12.379 7fdf709222e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:29:12.379 7fdf709222e8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:29:12.380 7fdf709222e8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:29:12.380 7fdf709222e8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:29:12.380 7fdf709222e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:29:12.380 7fdf7089c390 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:29:12.381 7fdf7089c390 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:29:12.609 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:13.610 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:14.611 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:15.612 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:16.613 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:17.615 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:18.616 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:19.617 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:20.618 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:21.619 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:22.620 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:23.620 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:24.621 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:25.622 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:26.623 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:27.624 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:28.624 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:29.626 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:30.627 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:31.628 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:32.629 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:33.630 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:34.630 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:35.631 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:36.632 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:37.634 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:38.635 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:39.635 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:40.636 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:41.637 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:42.638 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:43.638 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:44.639 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:45.641 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:46.642 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:47.643 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:48.644 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:49.645 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:50.646 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:51.646 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:52.648 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:53.649 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:54.650 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:55.651 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:56.652 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:57.654 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:58.654 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:59.655 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:00.657 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:01.658 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:02.658 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:03.659 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:04.660 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:05.661 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:06.293 7fdf70897f60 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:30:06.662 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:07.663 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:08.664 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:09.051 7fdf79f5cd08 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 5ba92628-0e8f-46ff-9c5b-baf2735173be 11/07/2024 02:30:09.207 7fdf79f5cd08 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 155e72c1-f8e6-4c46-b897-708544212d8a 11/07/2024 02:30:09.685 7fdf7144ee10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:53.708 7fc182427488 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:30:53.709 7fc182427488 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:30:53.709 7fc182427488 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:30:53.709 7fc179af96d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:30:53.709 7fc179af96d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:30:53.710 7fc179af96d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:30:53.710 7fc179af96d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:30:53.710 7fc179af96d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.710 7fc179af96d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:30:53.710 7fc179af96d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.710 7fc179af96d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.710 7fc179af96d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 113 start_time = 1730946653710044 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:30:53.710 7fc179af96d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.745 7fc182427488 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 11/07/2024 02:30:53.751 7fc182427488 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:30:53.751 7fc179a4a240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.751 7fc179a4a240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:30:53.751 7fc179a4a240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.751 7fc179a4a240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.751 7fc179a4a240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:30:53.752 7fc179a4a240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.755 7fc182427488 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:30:53.755 7fc179a4a0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.756 7fc179a4a0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:30:53.756 7fc179a4a0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.762 7fc182427488 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:30:53.762 7fc1799a12e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.762 7fc1799a12e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:30:53.762 7fc1799a12e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.765 7fc182427488 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:30:53.765 7fc1799a12e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.765 7fc1799a12e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:30:53.765 7fc1799a12e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.774 7fc182427488 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:30:53.774 7fc182427488 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:30:53.775 7fc182427488 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:30:53.775 7fc182427488 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:30:53.775 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:30:53.775 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:30:53.776 7fc17c50b0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:30:53.776 7fc17c50b0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:30:53.776 7fc1799a1cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.776 7fc1799a1cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:30:53.776 7fc1799a1cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.778 7fc182427488 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:30:53.781 7fc17c50b0f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:30:53.783 7fc1820ca6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:30:53.783 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:30:53.800 7fc182427488 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:30:53.802 7fc1790418d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.802 7fc1790418d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:30:53.802 7fc1790418d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.807 7fc182427488 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:30:53.807 7fc1790e4828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.807 7fc1790e4828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:30:53.807 7fc1790e4828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.811 7fc182427488 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:30:53.822 7fc182427488 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:30:53.823 7fc1790792e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.823 7fc1790792e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:30:53.823 7fc1790792e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.835 7fc179963048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.836 7fc179963048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:30:53.836 7fc179963048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.843 7fc1799a1a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.843 7fc1799a1a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:30:53.843 7fc1799a1a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:53.843 7fc182427488 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:30:53.844 7fc182427488 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:30:53.844 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:30:53.844 7fc182427488 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1. 11/07/2024 02:30:53.849 7fc1820ca6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:30:53.855 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:30:53.863 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:30:53.866 7fc1820ca6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:30:53.871 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:30:54.356 7fc182427488 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:30:54.621 7fc182427488 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:30:54.821 7fc182427488 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue 11/07/2024 02:30:54.821 7fc182427488 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue 11/07/2024 02:30:54.873 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:54.881 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:30:54.888 7fc1820ca6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:30:54.896 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue'] 11/07/2024 02:30:55.322 7fc182427488 [contrail-device-manager] [WARNING]: Process 113 prepared to run in Full mode... 11/07/2024 02:30:55.322 7fc182427488 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:30:55.322 7fc182427488 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:30:55.325 7fc182427488 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:30:55.325 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:30:55.326 7fc182427488 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:30:55.326 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:30:55.327 7fc182427488 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:30:55.327 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:30:55.331 7fc182427488 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:30:55.331 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:30:55.331 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:30:55.331 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:30:55.331 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:30:55.331 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:30:55.331 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:30:55.331 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:30:55.331 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:30:55.331 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:30:55.332 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:30:55.332 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:30:55.332 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:30:55.333 7fc182427488 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:30:55.333 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:30:55.333 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:30:55.333 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:30:55.334 7fc182427488 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:30:55.334 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:30:55.334 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:30:55.334 7fc182427488 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:30:55.334 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:30:55.335 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:30:55.335 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:30:55.335 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:30:55.335 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:30:55.336 7fc182427488 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:30:55.336 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:30:55.337 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:30:55.337 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:30:55.337 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:30:55.337 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:30:55.337 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:30:55.337 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:30:55.337 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:30:55.337 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:30:55.337 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:30:55.338 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:30:55.338 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:30:55.338 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:30:55.338 7fc182427488 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:30:55.340 7fc182427488 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:30:55.381 7fc178d980f0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:30:55.381 7fc178d980f0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:30:55.381 7fc178d8f828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:55.381 7fc178d8f828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:30:55.381 7fc178d8f828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:55.395 7fc178dd2ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:55.396 7fc178dd2ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:30:55.396 7fc178dd2ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:55.899 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:55.909 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:30:55.912 7fc1820ca6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:30:55.921 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:30:56.924 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:57.710 7fc1785fca20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:30:57.711 7fc1785fca20 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:30:57.711 7fc1785fca20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:30:57.711 7fc1785fca20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:30:57.711 7fc1785fca20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:30:57.711 7fc1785fca20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:30:57.712 7fc1785fca20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:30:57.712 7fc1785fca20 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:30:57.712 7fc1785fca20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:30:57.712 7fc1785fca20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:30:57.712 7fc1785fca20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:30:57.712 7fc1785c8c18 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:30:57.713 7fc1785c8c18 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:30:57.715 7fc1785c8198 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 11/07/2024 02:30:57.716 7fc1785c8198 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:30:57.716 7fc17a6a5eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:30:57.716 7fc17a6a5eb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:30:57.716 7fc17a6a5eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:30:57.716 7fc17a6a5eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:30:57.717 7fc17a6a5eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:30:57.717 7fc1785c8978 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:30:57.718 7fc1785c8978 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:30:57.925 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:58.926 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:59.927 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:00.927 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:01.929 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:02.930 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:03.931 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:04.932 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:05.933 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:06.934 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:07.935 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:08.936 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:09.937 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:10.937 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:11.939 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:12.943 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:13.945 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:14.947 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:15.948 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:16.949 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:17.949 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:18.950 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:19.951 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:20.952 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:21.952 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:22.953 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:23.955 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:24.956 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:25.956 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:26.957 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:27.958 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:28.960 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:29.961 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:30.962 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:31.962 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:32.964 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:33.964 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:34.965 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:35.966 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:36.968 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:37.969 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:38.970 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:39.971 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:40.972 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:41.973 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:42.974 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:43.974 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:44.975 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:45.976 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:46.977 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:47.978 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:48.978 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:49.980 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:50.982 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:51.983 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:52.984 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:53.985 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:54.986 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:55.759 7fc182427488 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 5ba92628-0e8f-46ff-9c5b-baf2735173be 11/07/2024 02:31:55.911 7fc182427488 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 155e72c1-f8e6-4c46-b897-708544212d8a 11/07/2024 02:31:55.988 7fc1820ca6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:39.673 7f5d1191af88 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:32:39.673 7f5d1191af88 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:32:39.673 7f5d1191af88 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:32:39.674 7f5d08fe96d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:32:39.674 7f5d08fe96d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:32:39.675 7f5d08fe96d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:32:39.675 7f5d08fe96d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:32:39.675 7f5d08fe96d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.675 7f5d08fe96d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:32:39.676 7f5d08fe96d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.676 7f5d08fe96d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.676 7f5d08fe96d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 113 start_time = 1730946759675119 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:32:39.676 7f5d08fe96d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.718 7f5d1191af88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 11/07/2024 02:32:39.723 7f5d1191af88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:32:39.723 7f5d08f38240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.724 7f5d08f38240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:32:39.724 7f5d08f38240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.724 7f5d08f38240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.724 7f5d08f38240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:32:39.724 7f5d08f38240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.733 7f5d1191af88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:32:39.733 7f5d08f380f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.734 7f5d08f380f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:32:39.734 7f5d08f380f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.739 7f5d1191af88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:32:39.740 7f5d08e912e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.740 7f5d08e912e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:32:39.740 7f5d08e912e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.743 7f5d1191af88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:32:39.743 7f5d08e912e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.743 7f5d08e912e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:32:39.743 7f5d08e912e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.750 7f5d1191af88 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:32:39.750 7f5d1191af88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:32:39.750 7f5d1191af88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:32:39.750 7f5d1191af88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:32:39.751 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:32:39.751 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:32:39.751 7f5d0b9fc0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:32:39.752 7f5d0b9fc0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:32:39.752 7f5d08e91cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.753 7f5d08e91cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:32:39.753 7f5d08e91cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.754 7f5d1191af88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:32:39.760 7f5d115c0780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:32:39.760 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:32:39.760 7f5d0b9fc0f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:32:39.780 7f5d1191af88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:32:39.781 7f5d085328d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.781 7f5d085328d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:32:39.781 7f5d085328d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.785 7f5d1191af88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:32:39.785 7f5d085d5828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.785 7f5d085d5828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:32:39.785 7f5d085d5828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.790 7f5d1191af88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:32:39.799 7f5d1191af88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:32:39.800 7f5d0856f2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.800 7f5d0856f2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:32:39.800 7f5d0856f2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.814 7f5d08e3ae10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.814 7f5d08e3ae10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:32:39.814 7f5d08e3ae10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.824 7f5d0859c198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.824 7f5d0859c198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:32:39.824 7f5d0859c198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.824 7f5d1191af88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:32:39.824 7f5d1191af88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:32:39.825 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:32:39.825 7f5d1191af88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1. 11/07/2024 02:32:39.827 7f5d115c0780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:32:39.833 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:32:39.841 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:32:39.844 7f5d115c0780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:32:39.854 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:32:40.337 7f5d1191af88 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:32:40.611 7f5d1191af88 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:32:40.789 7f5d1191af88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue 11/07/2024 02:32:40.790 7f5d1191af88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue 11/07/2024 02:32:40.855 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:40.865 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:32:40.870 7f5d115c0780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:32:40.879 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue'] 11/07/2024 02:32:41.291 7f5d1191af88 [contrail-device-manager] [WARNING]: Process 113 prepared to run in Full mode... 11/07/2024 02:32:41.291 7f5d1191af88 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:32:41.291 7f5d1191af88 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:32:41.294 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:32:41.294 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:32:41.295 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:32:41.295 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:32:41.296 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:32:41.296 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:32:41.300 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:32:41.300 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:32:41.301 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:32:41.301 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:32:41.301 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:32:41.301 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:32:41.301 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:32:41.301 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:32:41.301 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:32:41.301 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:32:41.301 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:32:41.301 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:32:41.301 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:32:41.303 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:32:41.303 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:32:41.303 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:32:41.303 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:32:41.304 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:32:41.304 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:32:41.304 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:32:41.304 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:32:41.304 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:32:41.304 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:32:41.305 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:32:41.305 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:32:41.305 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:32:41.306 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:32:41.306 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:32:41.306 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:32:41.306 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:32:41.306 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:32:41.306 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:32:41.306 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:32:41.306 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:32:41.306 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:32:41.306 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:32:41.306 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:32:41.307 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:32:41.307 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:32:41.307 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:32:41.307 7f5d1191af88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:32:41.310 7f5d1191af88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:32:41.349 7f5d083fb780 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:32:41.349 7f5d083fb780 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:32:41.350 7f5d08261978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:41.350 7f5d08261978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:32:41.350 7f5d08261978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:41.364 7f5d08218048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:41.365 7f5d08218048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:32:41.365 7f5d08218048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:41.883 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:41.891 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:32:41.895 7f5d115c0780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:32:41.911 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:32:42.914 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:43.675 7f5d03a97198 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:32:43.678 7f5d03a97198 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:32:43.678 7f5d03a97198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:32:43.678 7f5d03a97198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:32:43.678 7f5d03a97198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:32:43.678 7f5d03a97198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:32:43.679 7f5d03a97198 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:32:43.679 7f5d03a97198 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:32:43.681 7f5d03a97198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:32:43.681 7f5d03a97198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:32:43.682 7f5d03a97198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:32:43.682 7f5d03acf978 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:32:43.685 7f5d03acf978 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:32:43.688 7f5d03acfb70 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 11/07/2024 02:32:43.688 7f5d03acfb70 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:32:43.689 7f5d03acfe10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:32:43.689 7f5d03acfe10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:32:43.689 7f5d03acfe10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:32:43.689 7f5d03acfe10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:32:43.689 7f5d03acfe10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:32:43.690 7f5d03a97ac8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:32:43.691 7f5d03a97ac8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:32:43.915 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:44.915 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:45.916 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:46.917 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:47.918 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:48.918 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:49.920 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:50.921 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:51.922 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:52.923 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:53.925 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:54.926 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:55.927 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:56.927 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:57.929 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:58.930 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:59.931 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:00.931 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:01.932 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:02.933 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:03.935 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:04.936 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:05.936 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:06.937 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:07.938 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:08.939 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:09.940 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:10.940 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:11.941 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:12.942 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:13.943 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:14.945 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:15.945 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:16.946 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:17.947 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:18.948 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:19.949 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:20.949 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:21.951 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:22.952 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:23.953 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:24.954 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:25.955 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:26.957 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:27.958 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:28.959 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:29.960 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:30.961 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:31.962 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:32.963 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:33.964 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:34.964 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:35.965 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:36.966 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:37.967 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:38.968 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:39.968 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:40.827 7f5d1191af88 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 5ba92628-0e8f-46ff-9c5b-baf2735173be 11/07/2024 02:33:40.969 7f5d115c0780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:40.979 7f5d1191af88 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 155e72c1-f8e6-4c46-b897-708544212d8a 11/07/2024 02:34:25.652 7f0e3d23de88 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:34:25.652 7f0e3d23de88 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:34:25.652 7f0e3d23de88 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:34:25.652 7f0e348d46d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:34:25.652 7f0e348d46d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:34:25.653 7f0e348d46d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:34:25.653 7f0e348d46d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:34:25.653 7f0e348d46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.653 7f0e348d46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:34:25.654 7f0e348d46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.654 7f0e348d46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.654 7f0e348d46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 113 start_time = 1730946865653073 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:34:25.654 7f0e348d46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.691 7f0e3d23de88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 11/07/2024 02:34:25.695 7f0e3d23de88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:34:25.696 7f0e34827240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.696 7f0e34827240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:34:25.696 7f0e34827240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.696 7f0e34827240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.696 7f0e34827240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:34:25.696 7f0e34827240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.699 7f0e3d23de88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:34:25.700 7f0e348270f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.700 7f0e348270f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:34:25.700 7f0e348270f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.706 7f0e3d23de88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:34:25.707 7f0e3477f2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.707 7f0e3477f2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:34:25.707 7f0e3477f2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.710 7f0e3d23de88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:34:25.711 7f0e3477f2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.711 7f0e3477f2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:34:25.711 7f0e3477f2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.718 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:34:25.719 7f0e3d23de88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:34:25.719 7f0e3d23de88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:34:25.719 7f0e3d23de88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:34:25.720 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:34:25.720 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:34:25.720 7f0e372e80f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:34:25.720 7f0e372e80f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:34:25.721 7f0e3477fcc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.721 7f0e3477fcc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:34:25.721 7f0e3477fcc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.722 7f0e3d23de88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:34:25.727 7f0e3ceabd68 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:34:25.727 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:34:25.727 7f0e372e80f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:34:25.767 7f0e3d23de88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:34:25.768 7f0e34620780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.769 7f0e34620780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:34:25.769 7f0e34620780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.778 7f0e3d23de88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:34:25.778 7f0e34624c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.778 7f0e34624c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:34:25.778 7f0e34624c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.782 7f0e3d23de88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:34:25.790 7f0e3d23de88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:34:25.790 7f0e346576d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.791 7f0e346576d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:34:25.791 7f0e346576d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.804 7f0e346886d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.805 7f0e346886d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:34:25.805 7f0e346886d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.811 7f0e39f56c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.811 7f0e39f56c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:34:25.811 7f0e39f56c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:25.811 7f0e3d23de88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:34:25.811 7f0e3d23de88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:34:25.811 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:34:25.813 7f0e3d23de88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1. 11/07/2024 02:34:25.836 7f0e3ceabd68 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:34:25.842 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:34:25.849 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:34:25.856 7f0e3ceabd68 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:34:25.861 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:34:26.333 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:34:26.598 7f0e3d23de88 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:34:26.791 7f0e3d23de88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue 11/07/2024 02:34:26.791 7f0e3d23de88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue 11/07/2024 02:34:26.862 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:26.872 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:34:26.876 7f0e3ceabd68 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:34:26.885 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue'] 11/07/2024 02:34:27.293 7f0e3d23de88 [contrail-device-manager] [WARNING]: Process 113 prepared to run in Full mode... 11/07/2024 02:34:27.293 7f0e3d23de88 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:34:27.293 7f0e3d23de88 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:34:27.296 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:34:27.296 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:34:27.297 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:34:27.297 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:34:27.298 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:34:27.298 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:34:27.303 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:34:27.303 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:34:27.303 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:34:27.303 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:34:27.303 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:34:27.303 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:34:27.303 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:34:27.303 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:34:27.304 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:34:27.304 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:34:27.304 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:34:27.304 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:34:27.304 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:34:27.305 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:34:27.305 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:34:27.305 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:34:27.306 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:34:27.306 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:34:27.306 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:34:27.306 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:34:27.307 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:34:27.307 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:34:27.307 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:34:27.307 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:34:27.307 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:34:27.307 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:34:27.308 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:34:27.308 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:34:27.308 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:34:27.308 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:34:27.308 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:34:27.308 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:34:27.308 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:34:27.308 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:34:27.309 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:34:27.309 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:34:27.309 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:34:27.309 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:34:27.309 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:34:27.309 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:34:27.309 7f0e3d23de88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:34:27.312 7f0e3d23de88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:34:27.353 7f0e344f16d8 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:34:27.353 7f0e344f16d8 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:34:27.353 7f0e342fdac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:27.354 7f0e342fdac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:34:27.354 7f0e342fdac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:27.368 7f0e34355ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:27.369 7f0e34355ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:34:27.369 7f0e34355ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:34:27.887 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:27.894 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:34:27.897 7f0e3ceabd68 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:34:27.922 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:34:28.925 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:29.653 7f0e340a4828 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:34:29.654 7f0e340a4828 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:34:29.654 7f0e340a4828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:34:29.654 7f0e340a4828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:34:29.655 7f0e340a4828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:34:29.655 7f0e340a4828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:34:29.655 7f0e340a4828 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:34:29.655 7f0e340a4828 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:34:29.655 7f0e340a4828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:34:29.655 7f0e340a4828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:34:29.655 7f0e340a4828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:34:29.656 7f0e340a4d68 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:34:29.657 7f0e340a4d68 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:34:29.659 7f0e340a4b70 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 11/07/2024 02:34:29.659 7f0e340a4b70 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:34:29.660 7f0e340a44e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:34:29.660 7f0e340a44e0 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:34:29.660 7f0e340a44e0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:34:29.660 7f0e340a44e0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:34:29.660 7f0e340a44e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:34:29.660 7f0e340a4c18 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:34:29.662 7f0e340a4c18 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:34:29.926 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:30.927 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:31.928 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:32.929 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:33.930 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:34.931 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:35.931 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:36.933 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:37.934 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:38.935 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:39.935 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:40.936 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:41.937 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:42.938 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:43.939 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:44.940 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:45.941 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:46.942 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:47.943 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:48.943 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:49.945 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:50.945 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:51.946 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:52.947 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:53.949 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:54.950 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:55.951 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:56.952 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:57.954 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:58.954 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:59.955 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:00.956 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:01.958 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:02.959 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:03.960 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:04.961 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:05.962 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:06.963 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:07.964 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:08.965 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:09.967 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:10.968 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:11.969 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:12.970 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:13.971 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:14.972 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:15.973 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:16.974 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:17.975 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:18.975 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:19.977 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:20.978 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:21.979 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:22.980 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:23.981 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:24.982 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:25.983 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:26.984 7f0e3ceabd68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:26.987 7f0e3d23de88 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 5ba92628-0e8f-46ff-9c5b-baf2735173be 11/07/2024 02:35:27.143 7f0e3d23de88 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 155e72c1-f8e6-4c46-b897-708544212d8a 11/07/2024 02:36:11.658 7f2e933a8b48 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:36:11.658 7f2e933a8b48 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:36:11.658 7f2e933a8b48 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:36:11.658 7f2e8aa746d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:36:11.659 7f2e8aa746d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:36:11.659 7f2e8aa746d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:36:11.659 7f2e8aa746d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:36:11.659 7f2e8aa746d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.660 7f2e8aa746d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:36:11.660 7f2e8aa746d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.660 7f2e8aa746d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.660 7f2e8aa746d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 113 start_time = 1730946971659666 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:36:11.660 7f2e8aa746d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.697 7f2e933a8b48 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 11/07/2024 02:36:11.702 7f2e933a8b48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:36:11.703 7f2e8a9c5240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.703 7f2e8a9c5240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:36:11.703 7f2e8a9c5240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.703 7f2e8a9c5240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.703 7f2e8a9c5240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:36:11.703 7f2e8a9c5240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.708 7f2e933a8b48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:36:11.709 7f2e8a9c50f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.709 7f2e8a9c50f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:36:11.710 7f2e8a9c50f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.716 7f2e933a8b48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:36:11.716 7f2e8a91e2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.716 7f2e8a91e2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:36:11.716 7f2e8a91e2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.720 7f2e933a8b48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:36:11.721 7f2e8a91e2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.721 7f2e8a91e2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:36:11.721 7f2e8a91e2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.730 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:36:11.730 7f2e933a8b48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:36:11.730 7f2e933a8b48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:36:11.730 7f2e933a8b48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:36:11.731 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:36:11.731 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:36:11.731 7f2e8d48b0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:36:11.731 7f2e8d48b0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:36:11.732 7f2e8a91ecc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.732 7f2e8a91ecc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:36:11.732 7f2e8a91ecc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.733 7f2e933a8b48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:36:11.737 7f2e90365780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:36:11.737 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:36:11.738 7f2e8d48b0f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:36:11.764 7f2e933a8b48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:36:11.766 7f2e89fbf8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.766 7f2e89fbf8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:36:11.766 7f2e89fbf8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.770 7f2e933a8b48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:36:11.770 7f2e8a063828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.770 7f2e8a063828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:36:11.770 7f2e8a063828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.774 7f2e933a8b48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:36:11.781 7f2e933a8b48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:36:11.782 7f2e89fd22e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.782 7f2e89fd22e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:36:11.782 7f2e89fd22e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.796 7f2e8a8c6c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.796 7f2e8a8c6c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:36:11.796 7f2e8a8c6c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.804 7f2e89fcfeb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.804 7f2e89fcfeb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:36:11.804 7f2e89fcfeb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:11.804 7f2e933a8b48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:36:11.804 7f2e933a8b48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:36:11.805 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:36:11.806 7f2e933a8b48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1. 11/07/2024 02:36:11.810 7f2e90365780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:36:11.820 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:36:11.827 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:36:11.830 7f2e90365780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:36:11.836 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:36:12.318 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:36:12.617 7f2e933a8b48 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:36:12.781 7f2e933a8b48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue 11/07/2024 02:36:12.781 7f2e933a8b48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue 11/07/2024 02:36:12.837 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:12.844 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:36:12.855 7f2e90365780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:36:12.863 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue'] 11/07/2024 02:36:13.282 7f2e933a8b48 [contrail-device-manager] [WARNING]: Process 113 prepared to run in Full mode... 11/07/2024 02:36:13.283 7f2e933a8b48 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:36:13.283 7f2e933a8b48 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:36:13.286 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:36:13.286 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:36:13.287 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:36:13.287 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:36:13.287 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:36:13.288 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:36:13.292 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:36:13.292 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:36:13.292 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:36:13.292 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:36:13.292 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:36:13.292 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:36:13.292 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:36:13.292 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:36:13.292 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:36:13.292 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:36:13.293 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:36:13.293 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:36:13.293 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:36:13.294 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:36:13.294 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:36:13.294 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:36:13.294 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:36:13.295 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:36:13.295 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:36:13.295 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:36:13.295 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:36:13.295 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:36:13.295 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:36:13.296 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:36:13.296 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:36:13.296 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:36:13.296 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:36:13.297 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:36:13.297 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:36:13.297 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:36:13.297 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:36:13.297 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:36:13.297 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:36:13.297 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:36:13.297 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:36:13.297 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:36:13.297 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:36:13.298 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:36:13.298 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:36:13.298 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:36:13.298 7f2e933a8b48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:36:13.301 7f2e933a8b48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:36:13.341 7f2e89e7fcc0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:36:13.341 7f2e89e7fcc0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:36:13.341 7f2e89ce6240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:13.342 7f2e89ce6240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:36:13.342 7f2e89ce6240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:13.356 7f2e89cefd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:13.356 7f2e89cefd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:36:13.356 7f2e89cefd68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:13.866 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:13.874 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:36:13.877 7f2e90365780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:36:13.889 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:36:14.892 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:15.659 7f2e8953ce10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:36:15.660 7f2e8953ce10 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:36:15.660 7f2e8953ce10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:36:15.660 7f2e8953ce10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:36:15.661 7f2e8953ce10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:36:15.661 7f2e8953ce10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:36:15.661 7f2e8953ce10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:36:15.661 7f2e8953ce10 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:36:15.661 7f2e8953ce10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:36:15.661 7f2e8953ce10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:36:15.661 7f2e8953ce10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:36:15.662 7f2e895566d8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:36:15.663 7f2e895566d8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:36:15.665 7f2e89556588 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 11/07/2024 02:36:15.665 7f2e89556588 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:36:15.666 7f2e8b624eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:36:15.666 7f2e8b624eb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:36:15.666 7f2e8b624eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:36:15.666 7f2e8b624eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:36:15.666 7f2e8b624eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:36:15.666 7f2e895562e8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:36:15.668 7f2e895562e8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:36:15.893 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:16.894 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:17.896 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:18.897 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:19.898 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:20.899 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:21.901 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:22.902 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:23.903 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:24.904 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:25.906 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:26.907 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:27.907 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:28.909 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:29.909 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:30.910 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:31.911 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:32.912 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:33.913 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:34.915 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:35.915 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:36.916 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:37.916 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:38.918 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:39.919 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:40.920 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:41.921 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:42.922 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:43.924 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:44.925 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:45.926 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:46.926 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:47.927 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:48.929 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:49.930 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:50.930 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:51.931 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:52.932 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:53.933 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:54.935 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:55.936 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:56.937 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:57.938 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:58.939 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:59.940 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:00.941 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:01.943 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:02.944 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:03.945 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:04.946 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:05.947 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:06.948 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:07.949 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:08.950 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:09.951 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:10.952 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:11.803 7f2e933a8b48 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 5ba92628-0e8f-46ff-9c5b-baf2735173be 11/07/2024 02:37:11.952 7f2e90365780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:11.955 7f2e933a8b48 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 155e72c1-f8e6-4c46-b897-708544212d8a 11/07/2024 02:37:57.707 7efe015db848 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:37:57.708 7efe015db848 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:37:57.708 7efe015db848 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:37:57.708 7efdf8c4a6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:37:57.708 7efdf8c4a6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:37:57.709 7efdf8c4a6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:37:57.709 7efdf8c4a6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:37:57.709 7efdf8c4a6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.709 7efdf8c4a6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:37:57.709 7efdf8c4a6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.709 7efdf8c4a6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.709 7efdf8c4a6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 113 start_time = 1730947077708948 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:37:57.709 7efdf8c4a6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.744 7efe015db848 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.22:5673//'] 11/07/2024 02:37:57.748 7efe015db848 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:37:57.749 7efdf8b98240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.749 7efdf8b98240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:37:57.749 7efdf8b98240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.749 7efdf8b98240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.749 7efdf8b98240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:37:57.749 7efdf8b98240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.761 7efe015db848 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:37:57.762 7efdf8b980f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.762 7efdf8b980f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:37:57.762 7efdf8b980f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.768 7efe015db848 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:37:57.768 7efdf8af12e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.768 7efdf8af12e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:37:57.768 7efdf8af12e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.780 7efe015db848 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:37:57.780 7efdf8af1240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.780 7efdf8af1240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:37:57.780 7efdf8af1240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.787 7efe015db848 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:37:57.787 7efe015db848 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:37:57.788 7efe015db848 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:37:57.788 7efe015db848 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:37:57.788 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:37:57.788 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:37:57.789 7efdfb65b0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:37:57.789 7efdfb65b0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:37:57.790 7efdf8af1cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.790 7efdf8af1cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:37:57.790 7efdf8af1cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.791 7efe015db848 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:37:57.795 7efdfb65b0f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:37:57.795 7efdfe537780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:37:57.795 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:37:57.823 7efe015db848 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:37:57.825 7efdf81928d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.825 7efdf81928d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:37:57.825 7efdf81928d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.829 7efe015db848 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:37:57.830 7efdf81db828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.830 7efdf81db828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:37:57.830 7efdf81db828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.833 7efe015db848 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:37:57.841 7efe015db848 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:37:57.841 7efdf81cb2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.841 7efdf81cb2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:37:57.842 7efdf81cb2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.855 7efdf8266048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.856 7efdf8266048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:37:57.856 7efdf8266048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.861 7efdf818e8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.862 7efdf818e8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:37:57.862 7efdf818e8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:57.862 7efe015db848 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:37:57.862 7efe015db848 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:37:57.862 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:37:57.863 7efe015db848 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1. 11/07/2024 02:37:57.865 7efdfe537780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:37:57.871 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:37:57.879 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:37:57.882 7efdfe537780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:37:57.887 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.'] 11/07/2024 02:37:58.374 7efe015db848 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:37:58.649 7efe015db848 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 11/07/2024 02:37:58.831 7efe015db848 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue 11/07/2024 02:37:58.831 7efe015db848 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue 11/07/2024 02:37:58.889 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:58.895 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:37:58.899 7efdfe537780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:37:58.907 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue'] 11/07/2024 02:37:59.332 7efe015db848 [contrail-device-manager] [WARNING]: Process 113 prepared to run in Full mode... 11/07/2024 02:37:59.332 7efe015db848 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:37:59.332 7efe015db848 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:37:59.336 7efe015db848 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:37:59.336 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:37:59.337 7efe015db848 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:37:59.337 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:37:59.338 7efe015db848 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:37:59.338 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:37:59.342 7efe015db848 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:37:59.342 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:37:59.342 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:37:59.342 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:37:59.342 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:37:59.342 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:37:59.343 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:37:59.343 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:37:59.343 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:37:59.343 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:37:59.343 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:37:59.343 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:37:59.343 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:37:59.344 7efe015db848 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:37:59.345 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:37:59.345 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:37:59.345 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:37:59.345 7efe015db848 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:37:59.345 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:37:59.345 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:37:59.346 7efe015db848 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:37:59.346 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:37:59.346 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:37:59.346 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:37:59.346 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:37:59.346 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:37:59.347 7efe015db848 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:37:59.347 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:37:59.347 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:37:59.348 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:37:59.348 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:37:59.348 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:37:59.348 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:37:59.348 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:37:59.348 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:37:59.348 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:37:59.348 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:37:59.348 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:37:59.349 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:37:59.349 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:37:59.349 7efe015db848 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:37:59.351 7efe015db848 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:37:59.393 7efdf804f4e0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:37:59.393 7efdf804f4e0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:37:59.393 7efdf3e20048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:59.394 7efdf3e20048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:37:59.394 7efdf3e20048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:59.408 7efdf3faff60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:59.408 7efdf3faff60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:37:59.409 7efdf3faff60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:37:59.911 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:59.918 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:37:59.921 7efdfe537780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:37:59.932 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2208-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2208-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:38:00.934 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:01.709 7efdf3710048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:38:01.711 7efdf3710048 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:38:01.711 7efdf3710048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:38:01.711 7efdf3710048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:38:01.711 7efdf3710048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2208-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.22:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.22:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.22:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.22:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:38:01.711 7efdf3710048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:38:01.711 7efdf3710048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:38:01.711 7efdf3710048 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:38:01.712 7efdf3710048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:38:01.712 7efdf3710048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:38:01.712 7efdf3710048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:38:01.712 7efdf37102e8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:38:01.713 7efdf37102e8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:38:01.718 7efdf3710240 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 11/07/2024 02:38:01.719 7efdf3710240 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:38:01.719 7efdf37108d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:38:01.719 7efdf37108d0 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:38:01.719 7efdf37108d0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:38:01.720 7efdf37108d0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:38:01.720 7efdf37108d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:38:01.720 7efdf3710d68 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:38:01.721 7efdf3710d68 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:38:01.935 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:02.935 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:03.936 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:04.937 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:05.937 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:06.938 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:07.941 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:08.942 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:09.943 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:10.944 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:11.945 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:12.946 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:13.947 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:14.948 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:15.949 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:16.950 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:17.951 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:18.952 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:19.953 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:20.955 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:21.955 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:22.957 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:23.959 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:24.960 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:26.000 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:26.496 7efdff9510f0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:38:27.001 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:28.002 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:29.003 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:30.004 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:31.006 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:32.007 7efdfe537780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process.