11/07/2024 02:07:46.686 7f3574e04588 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:07:46.687 7f3574e04588 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:07:46.687 7f3574e04588 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:07:46.688 7f356c45e6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:07:46.688 7f356c45e6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:07:46.690 7f356c45e6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:07:46.691 7f356c45e6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:07:46.691 7f356c45e6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:07:46.691 7f356c45e6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:07:46.691 7f356c45e6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:07:46.691 7f356c45e6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:07:46.692 7f356c45e6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 119 start_time = 1730945266690435 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:07:46.692 7f356c45e6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:07:46.773 7f3574e04588 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:07:46.792 7f3574e04588 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:07:46.793 7f356c3ec630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:07:46.795 7f356c3ec630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:07:46.795 7f356c3ec630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:07:46.795 7f356c3ec630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:07:46.795 7f356c3ec630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:07:46.795 7f356c3ec630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:07:50.689 7f3571d4d780 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:07:50.691 7f3571d4d780 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:07:50.691 7f3571d4d780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:07:50.691 7f3571d4d780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:07:50.692 7f3571d4d780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:07:50.692 7f3571d4d780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:07:50.692 7f3571d4d780 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:07:50.692 7f3571d4d780 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:07:50.693 7f3571d4d780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:07:50.693 7f3571d4d780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:07:50.693 7f3571d4d780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:07:50.693 7f356c304588 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:07:50.694 7f356c304588 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:07:50.697 7f356c3040f0 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:07:50.698 7f356c3040f0 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:07:50.698 7f356d00ceb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:07:50.698 7f356d00ceb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:07:50.699 7f356d00ceb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:07:50.699 7f356d00ceb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:07:50.699 7f356d00ceb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:07:50.700 7f356c304780 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:07:50.700 7f356c304780 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:07:52.167 7f356c3040f0 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001290 11/07/2024 02:07:52.168 7f356c3040f0 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:07:52.168 7f357ad0b978 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 11/07/2024 02:07:52.168 7f357ad0b978 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 2 11/07/2024 02:07:52.168 7f357ad0b978 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:07:52.169 7f357ad0b978 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:08:16.849 7f3574e04588 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:08:18.082 7f3574e04588 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:08:18.088 7f3574e04588 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:08:18.842 7f3574e04588 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:08:18.843 7f3574e04588 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:08:18.843 7f3574e04588 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:08:18.843 7f3574e04588 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:08:18.844 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:08:18.844 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:08:18.846 7f356d078d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:08:18.850 7f356d078d68 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:08:18.852 7f3574e04588 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:08:18.871 7f356d078d68 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:08:18.872 7f3574c9e630 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:08:18.872 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:08:18.906 7f3574e04588 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:08:19.056 7f3574e04588 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:08:19.065 7f3574e04588 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:08:19.090 7f3574e04588 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:08:19.110 7f3574e04588 [contrail-device-manager] [ERROR]: Error in db walk read 'type' 11/07/2024 02:08:19.144 7f3574e04588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:08:19.144 7f3574e04588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:08:19.144 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:08:19.150 7f3574e04588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:08:19.158 7f3574c9e630 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:08:19.208 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:08:19.229 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:08:19.252 7f3574c9e630 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:08:19.276 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:08:19.664 7f3574e04588 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:08:20.225 7f3574e04588 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:08:20.279 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:20.655 7f3574e04588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:08:20.656 7f3574e04588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:08:21.157 7f3574e04588 [contrail-device-manager] [WARNING]: Process 119 prepared to run in Full mode... 11/07/2024 02:08:21.158 7f3574e04588 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:08:21.158 7f3574e04588 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:08:21.162 7f3574e04588 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:08:21.162 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:08:21.163 7f3574e04588 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:08:21.163 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:08:21.165 7f3574e04588 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:08:21.165 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:08:21.170 7f3574e04588 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:08:21.170 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:08:21.171 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:08:21.174 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:08:21.174 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:08:21.174 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:08:21.174 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:08:21.176 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:08:21.176 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:08:21.176 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:08:21.176 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:08:21.176 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:08:21.176 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:08:21.183 7f3574e04588 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:08:21.183 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:08:21.183 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:08:21.183 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:08:21.187 7f3574e04588 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:08:21.187 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:08:21.187 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:08:21.189 7f3574e04588 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:08:21.189 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:08:21.189 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:08:21.189 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:08:21.189 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:08:21.189 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:08:21.196 7f3574e04588 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:08:21.198 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:08:21.198 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:08:21.198 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:08:21.198 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:08:21.198 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:08:21.198 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:08:21.199 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:08:21.199 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:08:21.199 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:08:21.199 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:08:21.199 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:08:21.199 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:08:21.199 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:08:21.199 7f3574e04588 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:08:21.208 7f3574e04588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:08:21.328 7f356c135828 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:08:21.328 7f356c135828 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:08:21.332 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:21.350 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:08:21.359 7f3574c9e630 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:08:21.399 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:08:22.403 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:23.405 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:24.407 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:25.409 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:26.411 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:28.044 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:29.045 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:30.046 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:31.054 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:32.055 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:33.061 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:34.064 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:35.065 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:36.066 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:37.067 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:38.069 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:39.070 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:40.073 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:41.074 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:42.075 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:43.076 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:44.077 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:45.085 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:46.086 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:47.087 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:48.088 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:49.089 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:50.090 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:51.091 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:52.092 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:53.093 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:54.095 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:55.096 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:56.098 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:57.099 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:58.101 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:08:59.101 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:00.102 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:01.103 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:02.104 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:03.106 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:04.107 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:05.108 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:06.110 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:07.111 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:08.112 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:09.113 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:10.116 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:11.117 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:12.118 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:13.120 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:14.125 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:15.127 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:16.226 7f3574c9e630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:57.922 7fcc92380a48 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:09:57.923 7fcc92380a48 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:09:57.924 7fcc92380a48 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:09:57.924 7fcc899ed6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:09:57.925 7fcc899ed6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:09:57.926 7fcc899ed6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:09:57.926 7fcc899ed6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:09:57.926 7fcc899ed6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:57.927 7fcc899ed6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:09:57.927 7fcc899ed6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:57.928 7fcc899ed6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:57.929 7fcc899ed6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 107 start_time = 1730945397926128 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:09:57.929 7fcc899ed6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:57.986 7fcc92380a48 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:09:57.998 7fcc92380a48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:09:57.998 7fcc8993d240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:57.999 7fcc8993d240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:09:57.999 7fcc8993d240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:57.999 7fcc8993d240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:57.999 7fcc8993d240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:09:57.999 7fcc8993d240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.008 7fcc92380a48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:09:58.009 7fcc8993d0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.009 7fcc8993d0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:09:58.009 7fcc8993d0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.021 7fcc92380a48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:09:58.022 7fcc898942e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.022 7fcc898942e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:09:58.022 7fcc898942e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.026 7fcc92380a48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:09:58.026 7fcc898942e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.026 7fcc898942e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:09:58.026 7fcc898942e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.035 7fcc92380a48 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:09:58.036 7fcc92380a48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:09:58.036 7fcc92380a48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:09:58.036 7fcc92380a48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:09:58.037 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:09:58.037 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:09:58.037 7fcc8c4000f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:09:58.037 7fcc8c4000f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:09:58.038 7fcc89894cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.038 7fcc89894cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:09:58.038 7fcc89894cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.040 7fcc92380a48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:09:58.046 7fcc91fc1c18 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:09:58.046 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:09:58.050 7fcc8c4000f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:09:58.077 7fcc92380a48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:09:58.079 7fcc89775780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.079 7fcc89775780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:09:58.080 7fcc89775780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.086 7fcc92380a48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:09:58.087 7fcc89777d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.087 7fcc89777d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:09:58.087 7fcc89777d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.102 7fcc92380a48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:09:58.120 7fcc92380a48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:09:58.121 7fcc897b9828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.122 7fcc897b9828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:09:58.122 7fcc897b9828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.159 7fcc8971ea20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.159 7fcc8971ea20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:09:58.160 7fcc8971ea20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.168 7fcc897478d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.168 7fcc897478d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:09:58.168 7fcc897478d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:58.170 7fcc92380a48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:09:58.170 7fcc92380a48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:09:58.170 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:09:58.172 7fcc92380a48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:09:58.176 7fcc91fc1c18 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:09:58.183 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:09:58.194 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:09:58.200 7fcc91fc1c18 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:09:58.209 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:09:58.684 7fcc92380a48 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:09:58.904 7fcc92380a48 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:09:59.211 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:59.317 7fcc92380a48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:09:59.318 7fcc92380a48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:09:59.819 7fcc92380a48 [contrail-device-manager] [WARNING]: Process 107 prepared to run in Full mode... 11/07/2024 02:09:59.819 7fcc92380a48 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:09:59.819 7fcc92380a48 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:09:59.823 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:09:59.823 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:09:59.824 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:09:59.824 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:09:59.826 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:09:59.826 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:09:59.838 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:09:59.839 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:09:59.839 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:09:59.839 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:09:59.839 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:09:59.839 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:09:59.839 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:09:59.840 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:09:59.840 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:09:59.840 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:09:59.840 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:09:59.840 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:09:59.840 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:09:59.841 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:09:59.842 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:09:59.842 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:09:59.842 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:09:59.843 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:09:59.843 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:09:59.843 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:09:59.843 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:09:59.843 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:09:59.843 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:09:59.844 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:09:59.844 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:09:59.844 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:09:59.844 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:09:59.845 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:09:59.845 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:09:59.845 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:09:59.845 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:09:59.845 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:09:59.845 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:09:59.845 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:09:59.845 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:09:59.845 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:09:59.845 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:09:59.846 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:09:59.846 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:09:59.846 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:09:59.846 7fcc92380a48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:09:59.855 7fcc92380a48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:09:59.960 7fcc89634c18 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:09:59.960 7fcc89634c18 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:09:59.960 7fcc88df3780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:59.970 7fcc88df3780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:09:59.970 7fcc88df3780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:10:00.019 7fcc88e4e828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:10:00.019 7fcc88e4e828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:10:00.019 7fcc88e4e828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:10:00.214 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:00.592 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:10:00.598 7fcc91fc1c18 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:10:00.659 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:10:01.663 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:01.926 7fcc88d28d68 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:10:02.563 7fcc88d28d68 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:10:02.563 7fcc88d28d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:10:02.564 7fcc88d28d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:10:02.564 7fcc88d28d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:10:02.564 7fcc88d28d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:10:02.564 7fcc88d28d68 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:10:02.564 7fcc88d28d68 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:10:02.565 7fcc88d28d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:10:02.566 7fcc88d28d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:10:02.566 7fcc88d28d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:10:02.566 7fcc88cc06d8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:10:02.567 7fcc88cc06d8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:10:02.586 7fcc88cc0588 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:10:02.587 7fcc88cc0588 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:10:02.588 7fcc88d60ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:10:02.588 7fcc88d60ac8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:10:02.588 7fcc88d60ac8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:10:02.588 7fcc88d60ac8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:10:02.588 7fcc88d60ac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:10:02.589 7fcc88cc0198 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:10:02.603 7fcc88cc0198 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:10:02.664 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:03.665 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:04.666 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:05.667 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:06.668 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:07.669 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:08.671 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:09.672 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:10.673 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:11.674 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:12.677 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:13.679 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:14.680 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:15.681 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:16.683 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:17.684 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:18.684 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:19.579 7fcc88d30f60 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:10:19.686 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:20.687 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:21.698 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:22.700 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:23.701 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:24.703 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:25.704 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:26.514 7fcc8989c630 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:10:26.705 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:27.706 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:28.708 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:29.821 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:30.823 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:31.681 7fcc89747630 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:10:31.824 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:32.825 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:33.828 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:34.829 7fcc91fc1c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:35.295 7fcc92380a48 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:10:35.382 7fcc92380a48 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:11:17.971 7f1507f73308 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:11:17.972 7f1507f73308 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:11:17.973 7f1507f73308 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:11:17.973 7f14ff6426d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:11:17.973 7f14ff6426d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:11:17.974 7f14ff6426d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:11:17.974 7f14ff6426d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:11:17.974 7f14ff6426d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:17.974 7f14ff6426d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:11:17.974 7f14ff6426d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:17.974 7f14ff6426d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:17.974 7f14ff6426d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1730945477974002 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:11:17.974 7f14ff6426d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.465 7f1507f73308 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:11:18.476 7f1507f73308 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:11:18.476 7f14ff594240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.477 7f14ff594240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:11:18.477 7f14ff594240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.477 7f14ff594240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.477 7f14ff594240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:11:18.477 7f14ff594240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.484 7f1507f73308 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:11:18.485 7f14ff5940f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.485 7f14ff5940f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:11:18.485 7f14ff5940f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.499 7f1507f73308 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:11:18.499 7f14ff4eb2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.500 7f14ff4eb2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:11:18.500 7f14ff4eb2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.504 7f1507f73308 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:11:18.506 7f14ff4eb240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.506 7f14ff4eb240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:11:18.506 7f14ff4eb240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.515 7f1507f73308 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:11:18.515 7f1507f73308 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:11:18.515 7f1507f73308 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:11:18.515 7f1507f73308 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:11:18.516 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:11:18.516 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:11:18.516 7f14ff58e048 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:11:18.516 7f14ff58e048 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:11:18.517 7f14ff4eb240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.517 7f14ff4eb240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:11:18.517 7f14ff4eb240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.519 7f1507f73308 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:11:18.524 7f1504f34780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:11:18.524 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:11:18.525 7f14ff58e048 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:11:18.549 7f1507f73308 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:11:18.550 7f14ff3cd780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.551 7f14ff3cd780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:11:18.551 7f14ff3cd780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.556 7f1507f73308 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:11:18.557 7f14ff3cfd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.557 7f14ff3cfd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:11:18.557 7f14ff3cfd68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.573 7f1507f73308 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:11:18.586 7f1507f73308 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:11:18.587 7f14ff40f828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.587 7f14ff40f828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:11:18.587 7f14ff40f828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.614 7f14ff37fa20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.614 7f14ff37fa20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:11:18.614 7f14ff37fa20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.632 7f14ff3a08d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.632 7f14ff3a08d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:11:18.632 7f14ff3a08d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:18.634 7f1507f73308 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:11:18.634 7f1507f73308 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:11:18.634 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:11:18.635 7f1507f73308 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:11:18.641 7f1504f34780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:11:18.664 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:11:18.680 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:11:18.686 7f1504f34780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:11:18.706 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:11:19.151 7f1507f73308 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:11:19.339 7f1507f73308 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:11:19.602 7f1507f73308 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:11:19.602 7f1507f73308 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:11:19.709 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:19.718 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:11:19.723 7f1504f34780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:11:19.736 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue'] 11/07/2024 02:11:20.104 7f1507f73308 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 11/07/2024 02:11:20.104 7f1507f73308 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:11:20.104 7f1507f73308 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:11:20.107 7f1507f73308 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:11:20.107 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:11:20.108 7f1507f73308 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:11:20.109 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:11:20.110 7f1507f73308 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:11:20.110 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:11:20.118 7f1507f73308 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:11:20.118 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:11:20.118 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:11:20.118 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:11:20.118 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:11:20.118 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:11:20.118 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:11:20.118 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:11:20.118 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:11:20.119 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:11:20.119 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:11:20.119 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:11:20.119 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:11:20.120 7f1507f73308 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:11:20.121 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:11:20.121 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:11:20.121 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:11:20.121 7f1507f73308 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:11:20.121 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:11:20.121 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:11:20.122 7f1507f73308 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:11:20.122 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:11:20.122 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:11:20.122 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:11:20.122 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:11:20.122 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:11:20.123 7f1507f73308 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:11:20.123 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:11:20.123 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:11:20.124 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:11:20.124 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:11:20.124 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:11:20.124 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:11:20.124 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:11:20.124 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:11:20.124 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:11:20.124 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:11:20.124 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:11:20.125 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:11:20.125 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:11:20.125 7f1507f73308 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:11:20.128 7f1507f73308 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:11:20.207 7f14ff293390 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:11:20.207 7f14ff293390 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:11:20.207 7f14feac3a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:20.207 7f14feac3a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:11:20.207 7f14feac3a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:20.244 7f14feaa6ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:20.245 7f14feaa6ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:11:20.245 7f14feaa6ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:11:20.746 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:20.756 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:11:20.760 7f1504f34780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:11:20.794 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:11:21.806 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:21.974 7f14fe10a828 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:11:21.975 7f14fe10a828 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:11:21.975 7f14fe10a828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:11:21.975 7f14fe10a828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:11:21.975 7f14fe10a828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:11:21.976 7f14fe10a828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:11:21.976 7f14fe10a828 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:11:21.976 7f14fe10a828 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:11:21.976 7f14fe10a828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:11:21.976 7f14fe10a828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:11:21.976 7f14fe10a828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:11:21.977 7f14fe1068d0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:11:21.978 7f14fe1068d0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:11:21.981 7f14fe106390 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:11:21.981 7f14fe106390 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:11:21.981 7f15001f0eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:11:21.982 7f15001f0eb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:11:21.982 7f15001f0eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:11:21.982 7f15001f0eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:11:21.982 7f15001f0eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:11:21.983 7f14fe106048 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:11:21.984 7f14fe106048 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:11:22.807 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:23.808 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:24.809 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:25.811 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:26.814 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:27.815 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:28.817 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:29.818 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:30.820 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:31.822 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:32.823 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:33.825 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:34.825 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:35.827 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:36.828 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:37.829 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:38.830 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:39.838 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:40.839 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:41.841 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:42.842 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:43.844 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:44.845 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:45.846 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:46.847 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:47.848 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:48.849 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:49.851 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:50.852 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:51.853 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:52.820 7f1507f73308 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:11:52.854 7f1504f34780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:52.923 7f1507f73308 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:12:38.060 7f10023e0708 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:12:38.061 7f10023e0708 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:12:38.061 7f10023e0708 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:12:38.061 7f0ff9a0d6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:12:38.061 7f0ff9a0d6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:12:38.063 7f0ff9a0d6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:12:38.063 7f0ff9a0d6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:12:38.063 7f0ff9a0d6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.063 7f0ff9a0d6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:12:38.063 7f0ff9a0d6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.063 7f0ff9a0d6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.063 7f0ff9a0d6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1730945558062855 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:12:38.063 7f0ff9a0d6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.103 7f10023e0708 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:12:38.110 7f10023e0708 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:12:38.111 7f0ff995d240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.112 7f0ff995d240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:12:38.112 7f0ff995d240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.112 7f0ff995d240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.112 7f0ff995d240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:12:38.112 7f0ff995d240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.116 7f10023e0708 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:12:38.117 7f0ff995d0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.117 7f0ff995d0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:12:38.117 7f0ff995d0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.124 7f10023e0708 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:12:38.126 7f0ff98b52e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.126 7f0ff98b52e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:12:38.126 7f0ff98b52e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.129 7f10023e0708 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:12:38.130 7f0ff98b5240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.130 7f0ff98b5240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:12:38.130 7f0ff98b5240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.145 7f10023e0708 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:12:38.145 7f10023e0708 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:12:38.145 7f10023e0708 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:12:38.146 7f10023e0708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:12:38.147 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:12:38.147 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:12:38.147 7f0ffc41c0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:12:38.148 7f0ffc41c0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:12:38.148 7f0ff98b5cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.148 7f0ff98b5cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:12:38.148 7f0ff98b5cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.151 7f10023e0708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:12:38.157 7f0ffc41c0f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:12:38.157 7f0fff2fc780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:12:38.157 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:12:38.185 7f10023e0708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:12:38.188 7f0ff9795780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.188 7f0ff9795780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:12:38.188 7f0ff9795780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.193 7f10023e0708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:12:38.193 7f0ff9797d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.194 7f0ff9797d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:12:38.194 7f0ff9797d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.199 7f10023e0708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:12:38.211 7f10023e0708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:12:38.212 7f0ff97dc828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.212 7f0ff97dc828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:12:38.212 7f0ff97dc828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.229 7f0ff9765a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.229 7f0ff9765a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:12:38.229 7f0ff9765a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.238 7f0ff97698d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.238 7f0ff97698d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:12:38.238 7f0ff97698d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:38.239 7f10023e0708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:12:38.240 7f10023e0708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:12:38.240 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:12:38.240 7f10023e0708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:12:38.246 7f0fff2fc780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:12:38.254 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:12:38.263 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:12:38.267 7f0fff2fc780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:12:38.277 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:12:38.754 7f10023e0708 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:12:38.979 7f10023e0708 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:12:39.279 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:39.860 7f10023e0708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:12:39.861 7f10023e0708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:12:40.364 7f10023e0708 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 11/07/2024 02:12:40.364 7f10023e0708 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:12:40.364 7f10023e0708 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:12:40.367 7f10023e0708 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:12:40.368 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:12:40.369 7f10023e0708 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:12:40.369 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:12:40.371 7f10023e0708 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:12:40.371 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:12:40.376 7f10023e0708 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:12:40.377 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:12:40.378 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:12:40.378 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:12:40.378 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:12:40.378 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:12:40.378 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:12:40.378 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:12:40.378 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:12:40.378 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:12:40.378 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:12:40.378 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:12:40.378 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:12:40.380 7f10023e0708 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:12:40.380 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:12:40.380 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:12:40.380 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:12:40.380 7f10023e0708 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:12:40.381 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:12:40.381 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:12:40.381 7f10023e0708 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:12:40.381 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:12:40.381 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:12:40.381 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:12:40.381 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:12:40.382 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:12:40.382 7f10023e0708 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:12:40.383 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:12:40.383 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:12:40.383 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:12:40.383 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:12:40.383 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:12:40.383 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:12:40.383 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:12:40.383 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:12:40.383 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:12:40.383 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:12:40.384 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:12:40.384 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:12:40.384 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:12:40.384 7f10023e0708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:12:40.387 7f10023e0708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:12:40.440 7f0ff96aacc0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:12:40.440 7f0ff96aacc0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:12:40.440 7f0ff8e12780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:40.441 7f0ff8e12780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:12:40.441 7f0ff8e12780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:40.466 7f0ff8e6f828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:40.466 7f0ff8e6f828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:12:40.466 7f0ff8e6f828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:40.597 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:40.618 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:12:40.633 7f0fff2fc780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:12:40.647 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:12:41.650 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:42.062 7f0ff84dc198 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:12:42.063 7f0ff84dc198 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:12:42.063 7f0ff84dc198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:12:42.063 7f0ff84dc198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:12:42.063 7f0ff84dc198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:12:42.063 7f0ff84dc198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:12:42.063 7f0ff84dc198 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:12:42.064 7f0ff84dc198 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:12:42.064 7f0ff84dc198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:12:42.064 7f0ff84dc198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:12:42.064 7f0ff84dc198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:12:42.064 7f0ff84dcc18 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:12:42.065 7f0ff84dcc18 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:12:42.069 7f0ff84dcac8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:12:42.070 7f0ff84dcac8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:12:42.070 7f0ff84dccc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:12:42.070 7f0ff84dccc0 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:12:42.070 7f0ff84dccc0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:12:42.070 7f0ff84dccc0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:12:42.071 7f0ff84dccc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:12:42.071 7f0ff84dc2e8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:12:42.072 7f0ff84dc2e8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:12:42.652 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:43.653 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:44.655 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:45.656 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:46.657 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:47.659 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:48.660 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:49.665 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:50.666 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:51.667 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:52.668 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:53.722 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:54.723 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:55.724 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:56.725 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:57.726 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:58.727 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:59.728 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:00.729 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:01.730 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:02.731 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:03.732 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:04.734 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:05.735 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:06.736 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:07.737 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:08.738 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:09.745 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:10.746 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:11.746 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:12.748 7f0fff2fc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:12.997 7f10023e0708 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:13:13.081 7f10023e0708 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:13:57.887 7f5d86e64748 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:13:57.887 7f5d86e64748 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:13:57.887 7f5d86e64748 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:13:57.887 7f5d7e5346d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:13:57.887 7f5d7e5346d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:13:57.888 7f5d7e5346d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:13:57.888 7f5d7e5346d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:13:57.888 7f5d7e5346d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.889 7f5d7e5346d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:13:57.889 7f5d7e5346d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.889 7f5d7e5346d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.889 7f5d7e5346d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 105 start_time = 1730945637888619 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:13:57.889 7f5d7e5346d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.927 7f5d86e64748 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:13:57.932 7f5d86e64748 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:13:57.933 7f5d7e484240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.933 7f5d7e484240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:13:57.933 7f5d7e484240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.933 7f5d7e484240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.933 7f5d7e484240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:13:57.933 7f5d7e484240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.937 7f5d86e64748 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:13:57.937 7f5d7e4840f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.938 7f5d7e4840f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:13:57.938 7f5d7e4840f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.946 7f5d86e64748 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:13:57.947 7f5d7e3dc2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.947 7f5d7e3dc2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:13:57.947 7f5d7e3dc2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.951 7f5d86e64748 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:13:57.952 7f5d7e3dc2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.952 7f5d7e3dc2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:13:57.952 7f5d7e3dc2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.968 7f5d86e64748 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:13:57.969 7f5d86e64748 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:13:57.969 7f5d86e64748 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:13:57.969 7f5d86e64748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:13:57.970 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:13:57.970 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:13:57.971 7f5d80f450f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:13:57.971 7f5d80f450f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:13:57.971 7f5d7e3dccc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.971 7f5d7e3dccc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:13:57.971 7f5d7e3dccc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.974 7f5d86e64748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:13:57.980 7f5d80f450f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:13:57.981 7f5d86b046d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:13:57.981 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:13:58.013 7f5d86e64748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:13:58.015 7f5d7e2bc780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:58.016 7f5d7e2bc780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:13:58.016 7f5d7e2bc780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:58.026 7f5d86e64748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:13:58.032 7f5d7e2bdd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:58.032 7f5d7e2bdd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:13:58.032 7f5d7e2bdd68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:58.038 7f5d86e64748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:13:58.057 7f5d86e64748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:13:58.059 7f5d7e301828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:58.059 7f5d7e301828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:13:58.059 7f5d7e301828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:58.104 7f5d7e28ea20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:58.105 7f5d7e28ea20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:13:58.105 7f5d7e28ea20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:58.118 7f5d7e2928d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:58.118 7f5d7e2928d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:13:58.118 7f5d7e2928d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:58.119 7f5d86e64748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:13:58.119 7f5d86e64748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:13:58.120 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:13:58.120 7f5d86e64748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:13:58.130 7f5d86b046d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:13:58.138 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:13:58.147 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:13:58.151 7f5d86b046d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:13:58.167 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:13:58.632 7f5d86e64748 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:13:58.796 7f5d86e64748 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:13:59.045 7f5d86e64748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:13:59.045 7f5d86e64748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:13:59.169 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:59.176 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:13:59.183 7f5d86b046d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:13:59.194 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue'] 11/07/2024 02:13:59.547 7f5d86e64748 [contrail-device-manager] [WARNING]: Process 105 prepared to run in Full mode... 11/07/2024 02:13:59.547 7f5d86e64748 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:13:59.547 7f5d86e64748 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:13:59.550 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:13:59.550 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:13:59.551 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:13:59.551 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:13:59.557 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:13:59.558 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:13:59.562 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:13:59.562 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:13:59.562 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:13:59.562 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:13:59.563 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:13:59.563 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:13:59.563 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:13:59.563 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:13:59.563 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:13:59.563 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:13:59.563 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:13:59.563 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:13:59.563 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:13:59.565 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:13:59.565 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:13:59.565 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:13:59.565 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:13:59.565 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:13:59.565 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:13:59.565 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:13:59.566 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:13:59.566 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:13:59.566 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:13:59.566 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:13:59.566 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:13:59.566 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:13:59.567 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:13:59.567 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:13:59.567 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:13:59.568 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:13:59.568 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:13:59.568 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:13:59.568 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:13:59.568 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:13:59.568 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:13:59.568 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:13:59.568 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:13:59.569 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:13:59.569 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:13:59.569 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:13:59.569 7f5d86e64748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:13:59.572 7f5d86e64748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:13:59.637 7f5d7e182828 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:13:59.637 7f5d7e182828 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:13:59.637 7f5d7d9b4a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:59.638 7f5d7d9b4a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:13:59.638 7f5d7d9b4a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:59.659 7f5d7d996ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:59.659 7f5d7d996ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:13:59.659 7f5d7d996ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:14:00.199 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:00.208 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:14:00.213 7f5d86b046d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:14:00.236 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:14:01.260 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:01.888 7f5d7d7f8d68 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:14:01.890 7f5d7d7f8d68 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:14:01.890 7f5d7d7f8d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:14:01.890 7f5d7d7f8d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:14:01.890 7f5d7d7f8d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:14:01.890 7f5d7d7f8d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:14:01.890 7f5d7d7f8d68 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:14:01.891 7f5d7d7f8d68 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:14:01.891 7f5d7d7f8d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:14:01.891 7f5d7d7f8d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:14:01.891 7f5d7d7f8d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:14:01.892 7f5d7d807630 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:14:01.894 7f5d7d807630 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:14:01.899 7f5d7d8074e0 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:14:01.900 7f5d7d8074e0 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:14:01.901 7f5d7f0e0eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:14:01.901 7f5d7f0e0eb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:14:01.902 7f5d7f0e0eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:14:01.902 7f5d7f0e0eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:14:01.902 7f5d7f0e0eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:14:01.903 7f5d7d807240 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:14:01.905 7f5d7d807240 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:14:02.261 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:03.263 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:04.264 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:05.265 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:06.266 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:07.267 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:08.268 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:09.269 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:10.270 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:11.271 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:12.272 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:13.273 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:14.274 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:15.281 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:16.282 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:17.283 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:18.284 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:19.285 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:20.286 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:21.287 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:22.288 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:23.288 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:24.289 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:25.290 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:26.292 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:27.294 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:27.322 7f5d7e292978 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:14:28.295 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:29.298 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:30.299 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:31.300 7f5d86b046d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:31.809 7f5d86e64748 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:14:31.887 7f5d7d8aceb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:14:31.904 7f5d86e64748 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:15:17.861 7f6822ed4f08 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:15:17.861 7f6822ed4f08 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:15:17.861 7f6822ed4f08 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:15:17.862 7f681a56c6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:15:17.862 7f681a56c6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:15:17.862 7f681a56c6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:15:17.862 7f681a56c6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:15:17.863 7f681a56c6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.863 7f681a56c6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:15:17.863 7f681a56c6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.863 7f681a56c6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.863 7f681a56c6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1730945717862760 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:15:17.863 7f681a56c6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.901 7f6822ed4f08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:15:17.906 7f6822ed4f08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:15:17.906 7f681a4bc240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.906 7f681a4bc240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:15:17.906 7f681a4bc240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.907 7f681a4bc240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.907 7f681a4bc240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:15:17.907 7f681a4bc240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.910 7f6822ed4f08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:15:17.910 7f681a4bc0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.911 7f681a4bc0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:15:17.911 7f681a4bc0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.917 7f6822ed4f08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:15:17.918 7f681a4132e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.918 7f681a4132e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:15:17.918 7f681a4132e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.921 7f6822ed4f08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:15:17.921 7f681a4132e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.922 7f681a4132e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:15:17.922 7f681a4132e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.929 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:15:17.930 7f6822ed4f08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:15:17.930 7f6822ed4f08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:15:17.930 7f6822ed4f08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:15:17.931 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:15:17.931 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:15:17.931 7f681cf810f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:15:17.932 7f681cf810f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:15:17.933 7f681a413cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.933 7f681a413cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:15:17.933 7f681a413cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.934 7f6822ed4f08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:15:17.941 7f681cf810f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:15:17.942 7f6822d9e4e0 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:15:17.942 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:15:17.966 7f6822ed4f08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:15:17.975 7f6819aef8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.975 7f6819aef8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:15:17.975 7f6819aef8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.980 7f6822ed4f08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:15:17.980 7f6819b3f978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.980 7f6819b3f978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:15:17.980 7f6819b3f978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.986 7f6822ed4f08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:15:17.996 7f6822ed4f08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:15:17.996 7f6819af0438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:17.996 7f6819af0438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:15:17.997 7f6819af0438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:18.015 7f681a3b8d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:18.015 7f681a3b8d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:15:18.015 7f681a3b8d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:18.022 7f6819b32588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:18.022 7f6819b32588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:15:18.022 7f6819b32588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:18.023 7f6822ed4f08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:15:18.023 7f6822ed4f08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:15:18.023 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:15:18.025 7f6822ed4f08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:15:18.028 7f6822d9e4e0 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:15:18.035 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:15:18.042 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:15:18.045 7f6822d9e4e0 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:15:18.051 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:15:18.537 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:15:18.694 7f6822ed4f08 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:15:19.054 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:20.056 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:20.318 7f6822ed4f08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:15:20.319 7f6822ed4f08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:15:20.820 7f6822ed4f08 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 11/07/2024 02:15:20.820 7f6822ed4f08 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:15:20.820 7f6822ed4f08 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:15:20.824 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:15:20.824 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:15:20.825 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:15:20.825 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:15:20.826 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:15:20.826 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:15:20.830 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:15:20.830 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:15:20.830 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:15:20.830 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:15:20.831 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:15:20.831 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:15:20.831 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:15:20.831 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:15:20.831 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:15:20.831 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:15:20.831 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:15:20.831 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:15:20.831 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:15:20.833 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:15:20.833 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:15:20.833 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:15:20.833 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:15:20.834 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:15:20.834 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:15:20.834 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:15:20.835 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:15:20.835 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:15:20.835 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:15:20.835 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:15:20.835 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:15:20.835 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:15:20.836 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:15:20.836 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:15:20.836 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:15:20.836 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:15:20.836 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:15:20.836 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:15:20.836 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:15:20.836 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:15:20.837 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:15:20.837 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:15:20.837 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:15:20.837 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:15:20.837 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:15:20.837 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:15:20.837 7f6822ed4f08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:15:20.840 7f6822ed4f08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:15:20.889 7f6819a0bcc0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:15:20.889 7f6819a0bcc0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:15:20.890 7f681916f630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:20.890 7f681916f630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:15:20.890 7f681916f630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:20.906 7f68191ca6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:20.907 7f68191ca6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:15:20.907 7f68191ca6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:21.057 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:21.066 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:15:21.071 7f6822d9e4e0 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:15:21.081 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:15:21.867 7f68190a8978 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:15:21.869 7f68190a8978 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:15:21.869 7f68190a8978 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:15:21.869 7f68190a8978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:15:21.870 7f68190a8978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:15:21.870 7f68190a8978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:15:21.870 7f68190a8978 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:15:21.871 7f68190a8978 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:15:21.872 7f68190a8978 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:15:21.872 7f68190a8978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:15:21.872 7f68190a8978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:15:21.872 7f68190374e0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:15:21.873 7f68190374e0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:15:21.878 7f6819037390 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:15:21.878 7f6819037390 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:15:21.879 7f681b118eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:15:21.879 7f681b118eb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:15:21.879 7f681b118eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:15:21.879 7f681b118eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:15:21.879 7f681b118eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:15:21.879 7f68190370f0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:15:21.883 7f68190370f0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:15:22.084 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:23.085 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:24.086 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:25.090 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:26.090 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:27.091 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:28.093 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:29.095 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:30.096 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:31.098 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:32.100 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:33.100 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:34.101 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:35.103 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:36.103 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:37.105 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:38.106 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:39.117 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:40.118 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:41.119 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:42.120 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:43.122 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:44.123 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:45.124 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:46.126 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:47.127 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:48.129 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:49.130 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:50.130 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:51.131 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:52.132 7f6822d9e4e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:52.522 7f6822ed4f08 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:15:52.596 7f6822ed4f08 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:16:37.877 7f898c8ffb88 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:16:37.877 7f898c8ffb88 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:16:37.877 7f898c8ffb88 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:16:37.878 7f89840436d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:16:37.878 7f89840436d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:16:37.878 7f89840436d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:16:37.879 7f89840436d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:16:37.879 7f89840436d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.879 7f89840436d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:16:37.879 7f89840436d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.879 7f89840436d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.879 7f89840436d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1730945797878782 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:16:37.879 7f89840436d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.918 7f898c8ffb88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:16:37.924 7f898c8ffb88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:16:37.930 7f897fed6240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.931 7f897fed6240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:16:37.931 7f897fed6240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.931 7f897fed6240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.931 7f897fed6240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:16:37.932 7f897fed6240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.936 7f898c8ffb88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:16:37.946 7f897fed60f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.946 7f897fed60f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:16:37.946 7f897fed60f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.947 7f898c8ffb88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:16:37.947 7f897fed6240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.947 7f897fed6240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:16:37.948 7f897fed6240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.952 7f898c8ffb88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:16:37.954 7f897fe2e390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.954 7f897fe2e390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:16:37.954 7f897fe2e390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.968 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:16:37.968 7f898c8ffb88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:16:37.969 7f898c8ffb88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:16:37.969 7f898c8ffb88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:16:37.969 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:16:37.970 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:16:37.971 7f89869a50f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:16:37.973 7f89869a50f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:16:37.973 7f897fe2ecc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.976 7f897fe2ecc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:16:37.976 7f897fe2ecc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:37.977 7f898c8ffb88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:16:37.984 7f8989882780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:16:37.984 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:16:37.985 7f89869a50f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:16:38.020 7f898c8ffb88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:16:38.021 7f897fd0f630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:38.022 7f897fd0f630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:16:38.022 7f897fd0f630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:38.028 7f898c8ffb88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:16:38.028 7f897fd616d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:38.028 7f897fd616d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:16:38.028 7f897fd616d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:38.037 7f898c8ffb88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:16:38.060 7f898c8ffb88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:16:38.061 7f897fd4f198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:38.061 7f897fd4f198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:16:38.061 7f897fd4f198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:38.090 7f897fcdf390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:38.090 7f897fcdf390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:16:38.090 7f897fcdf390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:38.105 7f897fce1240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:38.106 7f897fce1240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:16:38.106 7f897fce1240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:38.106 7f898c8ffb88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:16:38.106 7f898c8ffb88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:16:38.106 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:16:38.107 7f898c8ffb88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:16:38.113 7f8989882780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:16:38.122 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:16:38.139 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:16:38.156 7f8989882780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:16:38.166 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:16:38.628 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:16:38.822 7f898c8ffb88 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:16:39.072 7f898c8ffb88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:16:39.072 7f898c8ffb88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:16:39.168 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:39.179 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:16:39.185 7f8989882780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:16:39.201 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue'] 11/07/2024 02:16:39.573 7f898c8ffb88 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 11/07/2024 02:16:39.573 7f898c8ffb88 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:16:39.573 7f898c8ffb88 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:16:39.584 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:16:39.584 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:16:39.590 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:16:39.591 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:16:39.595 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:16:39.595 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:16:39.609 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:16:39.609 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:16:39.610 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:16:39.610 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:16:39.610 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:16:39.610 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:16:39.610 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:16:39.610 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:16:39.610 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:16:39.611 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:16:39.611 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:16:39.611 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:16:39.611 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:16:39.615 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:16:39.616 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:16:39.616 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:16:39.616 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:16:39.617 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:16:39.617 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:16:39.618 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:16:39.618 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:16:39.618 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:16:39.619 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:16:39.619 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:16:39.619 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:16:39.619 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:16:39.621 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:16:39.621 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:16:39.621 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:16:39.621 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:16:39.621 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:16:39.621 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:16:39.621 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:16:39.622 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:16:39.622 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:16:39.622 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:16:39.622 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:16:39.623 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:16:39.623 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:16:39.623 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:16:39.624 7f898c8ffb88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:16:39.629 7f898c8ffb88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:16:39.702 7f897fc24ac8 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:16:39.702 7f897fc24ac8 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:16:39.703 7f897f3fbe10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:39.703 7f897f3fbe10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:16:39.703 7f897f3fbe10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:39.729 7f897f3e9eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:39.730 7f897f3e9eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:16:39.730 7f897f3e9eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:16:40.205 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:40.228 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:16:40.243 7f8989882780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:16:40.268 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:16:41.272 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:41.879 7f897f251d68 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:16:41.880 7f897f251d68 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:16:41.880 7f897f251d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:16:41.880 7f897f251d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:16:41.880 7f897f251d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:16:41.881 7f897f251d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:16:41.881 7f897f251d68 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:16:41.881 7f897f251d68 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:16:41.881 7f897f251d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:16:41.881 7f897f251d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:16:41.881 7f897f251d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:16:41.882 7f897f2c6a20 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:16:41.895 7f897f2c6a20 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:16:41.930 7f897f2c68d0 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:16:41.931 7f897f2c68d0 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:16:41.931 7f897fce70f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:16:41.931 7f897fce70f0 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:16:41.932 7f897fce70f0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:16:41.932 7f897fce70f0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:16:41.932 7f897fce70f0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:16:41.932 7f897f2c64e0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:16:41.934 7f897f2c64e0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:16:42.273 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:43.274 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:44.275 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:45.278 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:46.279 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:47.280 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:48.282 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:49.283 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:50.284 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:51.285 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:52.287 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:53.289 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:54.290 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:55.291 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:56.292 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:57.294 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:58.959 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:59.960 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:00.962 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:01.963 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:02.964 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:03.965 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:04.966 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:05.967 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:06.968 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:07.970 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:08.972 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:09.974 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:10.976 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:11.978 7f8989882780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:12.561 7f898c8ffb88 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:17:12.641 7f898c8ffb88 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:17:57.907 7f56a4662c88 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:17:57.907 7f56a4662c88 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:17:57.907 7f56a4662c88 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:17:57.907 7f5697cf46d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:17:57.907 7f5697cf46d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:17:57.908 7f5697cf46d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:17:57.908 7f5697cf46d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:17:57.908 7f5697cf46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:57.908 7f5697cf46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:17:57.908 7f5697cf46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:57.908 7f5697cf46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:57.908 7f5697cf46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 105 start_time = 1730945877908212 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:17:57.908 7f5697cf46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:57.968 7f56a4662c88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:17:57.990 7f56a4662c88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:17:57.991 7f5697c44240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:57.991 7f5697c44240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:17:57.991 7f5697c44240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:57.991 7f5697c44240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:57.991 7f5697c44240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:17:57.991 7f5697c44240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.006 7f56a4662c88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:17:58.006 7f5697c440f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.007 7f5697c440f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:17:58.007 7f5697c440f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.021 7f56a4662c88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:17:58.022 7f5697b9d2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.022 7f5697b9d2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:17:58.023 7f5697b9d2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.026 7f56a4662c88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:17:58.026 7f5697b9d2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.026 7f5697b9d2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:17:58.026 7f5697b9d2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.034 7f56a4662c88 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:17:58.034 7f56a4662c88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:17:58.035 7f56a4662c88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:17:58.035 7f56a4662c88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:17:58.035 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:17:58.035 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:17:58.036 7f569e71d0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:17:58.036 7f569e71d0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:17:58.036 7f5697b9dcc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.036 7f5697b9dcc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:17:58.036 7f5697b9dcc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.037 7f56a4662c88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:17:58.044 7f569e71d0f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:17:58.045 7f56a4548630 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:17:58.046 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:17:58.072 7f56a4662c88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:17:58.074 7f5697a7c780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.074 7f5697a7c780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:17:58.074 7f5697a7c780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.081 7f56a4662c88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:17:58.081 7f5697a7fd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.081 7f5697a7fd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:17:58.081 7f5697a7fd68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.088 7f56a4662c88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:17:58.101 7f56a4662c88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:17:58.101 7f5697abc828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.102 7f5697abc828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:17:58.102 7f5697abc828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.127 7f5697a50a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.127 7f5697a50a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:17:58.127 7f5697a50a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.135 7f5697a538d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.135 7f5697a538d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:17:58.135 7f5697a538d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:58.138 7f56a4662c88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:17:58.138 7f56a4662c88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:17:58.138 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:17:58.139 7f56a4662c88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:17:58.153 7f56a4548630 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:17:58.164 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:17:58.175 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:17:58.180 7f56a4548630 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:17:58.189 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:17:58.649 7f56a4662c88 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:17:58.807 7f56a4662c88 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:17:59.192 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:59.764 7f56a4662c88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:17:59.764 7f56a4662c88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:18:00.194 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:00.206 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:18:00.211 7f56a4548630 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:18:00.222 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue'] 11/07/2024 02:18:00.265 7f56a4662c88 [contrail-device-manager] [WARNING]: Process 105 prepared to run in Full mode... 11/07/2024 02:18:00.265 7f56a4662c88 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:18:00.265 7f56a4662c88 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:18:00.269 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:18:00.269 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:18:00.270 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:18:00.270 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:18:00.271 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:18:00.271 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:18:00.276 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:18:00.276 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:18:00.276 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:18:00.276 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:18:00.277 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:18:00.277 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:18:00.277 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:18:00.277 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:18:00.277 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:18:00.277 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:18:00.277 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:18:00.277 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:18:00.277 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:18:00.279 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:18:00.279 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:18:00.279 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:18:00.279 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:18:00.279 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:18:00.280 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:18:00.280 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:18:00.280 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:18:00.280 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:18:00.280 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:18:00.280 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:18:00.281 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:18:00.281 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:18:00.282 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:18:00.282 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:18:00.282 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:18:00.282 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:18:00.283 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:18:00.283 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:18:00.283 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:18:00.283 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:18:00.283 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:18:00.283 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:18:00.283 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:18:00.283 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:18:00.284 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:18:00.284 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:18:00.284 7f56a4662c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:18:00.287 7f56a4662c88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:18:00.353 7f5697942ac8 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:18:00.354 7f5697942ac8 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:18:00.354 7f5697175a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:18:00.354 7f5697175a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:18:00.355 7f5697175a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:18:00.392 7f5697157ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:18:00.392 7f5697157ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:18:00.392 7f5697157ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:18:01.224 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:01.234 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:18:01.239 7f56a4548630 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:18:01.249 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:18:01.908 7f5696fb9cc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:18:01.909 7f5696fb9cc0 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:18:01.910 7f5696fb9cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:18:01.910 7f5696fb9cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:18:01.910 7f5696fb9cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:18:01.910 7f5696fb9cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:18:01.910 7f5696fb9cc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:18:01.910 7f5696fb9cc0 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:18:01.911 7f5696fb9cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:18:01.911 7f5696fb9cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:18:01.911 7f5696fb9cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:18:01.911 7f5696fc4630 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:18:01.912 7f5696fc4630 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:18:01.917 7f5696fc44e0 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:18:01.917 7f5696fc44e0 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:18:01.918 7f569c8b9eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:18:01.918 7f569c8b9eb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:18:01.918 7f569c8b9eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:18:01.918 7f569c8b9eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:18:01.918 7f569c8b9eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:18:01.919 7f5696fc4240 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:18:01.920 7f5696fc4240 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:18:02.251 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:03.252 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:04.253 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:05.254 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:06.255 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:07.255 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:08.257 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:09.258 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:10.260 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:11.606 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:12.607 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:13.610 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:14.611 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:15.612 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:16.622 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:17.624 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:18.626 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:19.795 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:20.796 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:21.799 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:22.800 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:23.801 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:24.801 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:25.802 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:26.803 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:27.804 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:28.805 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:29.815 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:30.816 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:31.818 7f56a4548630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:31.933 7f56a4662c88 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:18:32.021 7f56a4662c88 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:19:18.439 7f69135fb2c8 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:19:18.439 7f69135fb2c8 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:19:18.439 7f69135fb2c8 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:19:18.440 7f690accb6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:19:18.440 7f690accb6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:19:18.440 7f690accb6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:19:18.441 7f690accb6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:19:18.441 7f690accb6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.441 7f690accb6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:19:18.441 7f690accb6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.441 7f690accb6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.441 7f690accb6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 105 start_time = 1730945958440781 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:19:18.441 7f690accb6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.485 7f69135fb2c8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:19:18.490 7f69135fb2c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:19:18.491 7f690ac1c240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.491 7f690ac1c240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:19:18.491 7f690ac1c240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.491 7f690ac1c240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.491 7f690ac1c240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:19:18.491 7f690ac1c240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.513 7f69135fb2c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:19:18.514 7f690ac1c240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.514 7f690ac1c240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:19:18.514 7f690ac1c240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.534 7f69135fb2c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:19:18.534 7f690ab732e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.535 7f690ab732e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:19:18.535 7f690ab732e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.551 7f69135fb2c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:19:18.552 7f690ab732e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.552 7f690ab732e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:19:18.552 7f690ab732e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.677 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:19:18.677 7f69135fb2c8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:19:18.678 7f69135fb2c8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:19:18.678 7f69135fb2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:19:18.679 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:19:18.679 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:19:18.680 7f690d6e10f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:19:18.680 7f690d6e10f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:19:18.681 7f690ab73cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.681 7f690ab73cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:19:18.681 7f690ab73cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.683 7f69135fb2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:19:18.720 7f690d6e10f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:19:18.720 7f690ab4de10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:19:18.720 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:19:18.723 7f69135fb2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:19:18.725 7f690aafcac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.726 7f690aafcac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:19:18.726 7f690aafcac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.774 7f69135fb2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:19:18.775 7f690aa5ed68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.775 7f690aa5ed68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:19:18.775 7f690aa5ed68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.781 7f69135fb2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:19:18.803 7f69135fb2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:19:18.805 7f690aaa0828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.806 7f690aaa0828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:19:18.806 7f690aaa0828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.833 7f690aaeea20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.833 7f690aaeea20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:19:18.833 7f690aaeea20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.884 7f690ab1eb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.884 7f690ab1eb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:19:18.885 7f690ab1eb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:18.885 7f69135fb2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:19:18.885 7f69135fb2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:19:18.886 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:19:18.887 7f69135fb2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:19:18.899 7f690ab4de10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:19:18.914 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:19:19.417 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:19:19.619 7f69135fb2c8 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:19:19.778 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:19:19.837 7f690ab4de10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:19:19.844 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:19:20.370 7f69135fb2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:19:20.370 7f69135fb2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:19:20.846 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:20.852 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:19:20.856 7f690ab4de10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:19:20.868 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue'] 11/07/2024 02:19:20.871 7f69135fb2c8 [contrail-device-manager] [WARNING]: Process 105 prepared to run in Full mode... 11/07/2024 02:19:20.871 7f69135fb2c8 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:19:20.871 7f69135fb2c8 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:19:20.875 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:19:20.875 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:19:20.876 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:19:20.876 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:19:20.878 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:19:20.878 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:19:20.882 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:19:20.882 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:19:20.882 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:19:20.882 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:19:20.882 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:19:20.882 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:19:20.882 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:19:20.882 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:19:20.882 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:19:20.882 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:19:20.883 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:19:20.883 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:19:20.883 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:19:20.884 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:19:20.885 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:19:20.885 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:19:20.885 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:19:20.885 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:19:20.885 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:19:20.885 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:19:20.886 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:19:20.886 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:19:20.886 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:19:20.886 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:19:20.886 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:19:20.886 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:19:20.887 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:19:20.887 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:19:20.887 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:19:20.887 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:19:20.887 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:19:20.887 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:19:20.887 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:19:20.888 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:19:20.888 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:19:20.888 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:19:20.888 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:19:20.888 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:19:20.888 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:19:20.888 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:19:20.888 7f69135fb2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:19:20.891 7f69135fb2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:19:20.960 7f690a174ac8 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:19:20.960 7f690a174ac8 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:19:20.961 7f6909941d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:20.961 7f6909941d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:19:20.961 7f6909941d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:21.013 7f690992ce10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:21.013 7f690992ce10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:19:21.014 7f690992ce10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:21.033 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:19:21.064 7f690ab4de10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:19:21.101 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:19:22.105 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:22.442 7f690980e4e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:19:22.443 7f690980e4e0 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:19:22.443 7f690980e4e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:19:22.443 7f690980e4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:19:22.444 7f690980e4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:19:22.444 7f690980e4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:19:22.444 7f690980e4e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:19:22.444 7f690980e4e0 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:19:22.444 7f690980e4e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:19:22.444 7f690980e4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:19:22.444 7f690980e4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:19:22.444 7f690980ef60 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:19:22.446 7f690980ef60 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:19:22.457 7f690980e6d8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:19:22.458 7f690980e6d8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:19:22.458 7f690980e048 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:19:22.459 7f690980e048 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:19:22.459 7f690980e048 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:19:22.459 7f690980e048 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:19:22.459 7f690980e048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:19:22.460 7f690980ed68 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:19:22.461 7f690980ed68 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:19:23.113 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:24.113 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:25.115 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:26.169 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:27.171 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:28.172 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:29.173 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:30.174 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:31.175 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:32.176 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:33.177 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:34.179 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:35.181 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:36.182 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:37.183 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:38.184 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:39.185 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:40.186 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:41.192 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:42.193 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:43.194 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:44.195 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:45.228 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:46.229 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:47.230 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:48.231 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:49.233 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:50.234 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:51.236 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:52.244 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:53.247 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:53.867 7f6910735240 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:19:54.248 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:55.262 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:56.263 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:56.841 7f69135fb2c8 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:19:57.320 7f690ab4de10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:57.379 7f69135fb2c8 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:20:40.194 7fba0aec0748 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:20:40.194 7fba0aec0748 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:20:40.195 7fba0aec0748 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:20:40.195 7fba0252c6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:20:40.195 7fba0252c6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:20:40.195 7fba0252c6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:20:40.196 7fba0252c6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:20:40.196 7fba0252c6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.196 7fba0252c6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:20:40.196 7fba0252c6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.196 7fba0252c6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.196 7fba0252c6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1730946040195865 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:20:40.197 7fba0252c6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.277 7fba0aec0748 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:20:40.288 7fba0aec0748 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:20:40.289 7fba0247d240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.289 7fba0247d240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:20:40.289 7fba0247d240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.289 7fba0247d240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.290 7fba0247d240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:20:40.290 7fba0247d240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.295 7fba0aec0748 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:20:40.295 7fba0247d0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.305 7fba0247d0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:20:40.306 7fba0247d0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.338 7fba0aec0748 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:20:40.340 7fba023d52e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.340 7fba023d52e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:20:40.340 7fba023d52e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.347 7fba0aec0748 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:20:40.348 7fba023d52e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.348 7fba023d52e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:20:40.348 7fba023d52e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.909 7fba0aec0748 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:20:40.911 7fba0aec0748 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:20:40.911 7fba0aec0748 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:20:40.912 7fba0aec0748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:20:40.912 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:20:40.912 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:20:40.912 7fba04f3e0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:20:40.913 7fba04f3e0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:20:40.913 7fba023d5cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.913 7fba023d5cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:20:40.913 7fba023d5cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.914 7fba0aec0748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:20:40.926 7fba0aafc588 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:20:40.926 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:20:40.926 7fba04f3e0f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:20:40.956 7fba0aec0748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:20:40.958 7fba01ab28d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.958 7fba01ab28d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:20:40.958 7fba01ab28d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.968 7fba0aec0748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:20:40.968 7fba01b05978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.969 7fba01b05978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:20:40.969 7fba01b05978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.975 7fba0aec0748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:20:40.991 7fba0aec0748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:20:40.993 7fba01af4438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:40.993 7fba01af4438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:20:40.993 7fba01af4438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:41.043 7fba01af48d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:41.043 7fba01af48d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:20:41.043 7fba01af48d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:41.055 7fba01a91780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:41.056 7fba01a91780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:20:41.056 7fba01a91780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:20:41.060 7fba0aec0748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:20:41.060 7fba0aec0748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:20:41.061 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:20:41.062 7fba0aec0748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:20:41.079 7fba0aafc588 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:20:41.096 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:20:41.112 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:20:41.118 7fba0aafc588 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:20:41.124 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:20:41.574 7fba0aec0748 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:20:41.775 7fba0aec0748 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:20:42.127 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:43.128 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:44.039 7fba0aec0748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:20:44.039 7fba0aec0748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:20:44.130 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:44.144 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:20:44.150 7fba0aafc588 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:20:44.168 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue'] 11/07/2024 02:20:44.196 7fba01a11ac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:20:44.197 7fba01a11ac8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:20:44.197 7fba01a11ac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:20:44.197 7fba01a11ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:20:44.198 7fba01a11ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:20:44.198 7fba01a11ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:20:44.198 7fba01a11ac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:20:44.199 7fba01a11ac8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:20:44.200 7fba01a11ac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:20:44.200 7fba01a11ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:20:44.200 7fba01a11ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:20:44.200 7fba06a98780 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:20:44.211 7fba06a98780 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:20:44.219 7fba02396048 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:20:44.222 7fba02396048 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:20:44.222 7fba0235e630 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:20:44.222 7fba0235e630 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:20:44.223 7fba0235e630 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:20:44.223 7fba0235e630 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:20:44.223 7fba0235e630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:20:44.223 7fba01a11ac8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:20:44.227 7fba01a11ac8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:20:44.540 7fba0aec0748 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 11/07/2024 02:20:44.540 7fba0aec0748 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:20:44.540 7fba0aec0748 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:20:44.544 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:20:44.544 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:20:44.546 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:20:44.547 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:20:44.548 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:20:44.548 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:20:44.556 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:20:44.556 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:20:44.556 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:20:44.557 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:20:44.557 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:20:44.557 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:20:44.557 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:20:44.557 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:20:44.557 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:20:44.557 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:20:44.557 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:20:44.557 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:20:44.557 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:20:44.559 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:20:44.559 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:20:44.559 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:20:44.559 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:20:44.560 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:20:44.560 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:20:44.560 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:20:44.561 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:20:44.561 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:20:44.561 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:20:44.561 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:20:44.562 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:20:44.562 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:20:44.563 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:20:44.563 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:20:44.563 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:20:44.563 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:20:44.563 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:20:44.564 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:20:44.564 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:20:44.564 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:20:44.564 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:20:44.564 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:20:44.564 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:20:44.564 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:20:44.564 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:20:44.564 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:20:44.564 7fba0aec0748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:20:44.567 7fba0aec0748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:20:44.672 7fba0197d438 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:20:44.672 7fba0197d438 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:20:45.172 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:45.182 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:20:45.188 7fba0aafc588 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:20:45.218 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:20:46.222 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:47.223 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:48.229 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:49.231 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:50.232 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:51.242 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:52.244 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:53.245 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:54.246 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:55.247 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:56.248 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:57.250 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:58.251 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:59.252 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:00.254 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:01.255 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:02.257 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:03.258 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:04.262 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:05.263 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:06.264 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:07.265 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:08.266 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:09.267 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:10.274 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:11.275 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:12.276 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:13.276 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:14.278 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:15.279 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:16.280 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:17.281 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:17.961 7fba0aec0748 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:21:18.058 7fba0aec0748 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:21:18.409 7fba0aafc588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:59.846 7f23069703c8 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:21:59.846 7f23069703c8 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:21:59.846 7f23069703c8 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:21:59.846 7f22fdfdd6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:21:59.847 7f22fdfdd6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:21:59.847 7f22fdfdd6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:21:59.847 7f22fdfdd6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:21:59.847 7f22fdfdd6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.848 7f22fdfdd6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:21:59.848 7f22fdfdd6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.848 7f22fdfdd6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.848 7f22fdfdd6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1730946119847581 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:21:59.848 7f22fdfdd6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.888 7f23069703c8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:21:59.893 7f23069703c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:21:59.894 7f22fdf2e240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.894 7f22fdf2e240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:21:59.894 7f22fdf2e240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.894 7f22fdf2e240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.894 7f22fdf2e240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:21:59.894 7f22fdf2e240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.903 7f23069703c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:21:59.904 7f22fdf2e240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.904 7f22fdf2e240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:21:59.904 7f22fdf2e240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.922 7f23069703c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:21:59.922 7f22fde852e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.923 7f22fde852e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:21:59.923 7f22fde852e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.938 7f23069703c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:21:59.939 7f22fde852e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.939 7f22fde852e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:21:59.939 7f22fde852e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.951 7f23069703c8 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:21:59.951 7f23069703c8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:21:59.951 7f23069703c8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:21:59.952 7f23069703c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:21:59.953 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:21:59.953 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:21:59.953 7f23009ee0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:21:59.953 7f23009ee0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:21:59.954 7f22fde85cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.954 7f22fde85cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:21:59.954 7f22fde85cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.955 7f23069703c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:21:59.962 7f22fde5fe10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:21:59.962 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:21:59.963 7f23009ee0f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:21:59.989 7f23069703c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:21:59.992 7f22fdd65780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.992 7f22fdd65780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:21:59.993 7f22fdd65780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:59.999 7f23069703c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:21:59.999 7f22fdd67d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:00.000 7f22fdd67d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:22:00.000 7f22fdd67d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:00.013 7f23069703c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:22:00.047 7f23069703c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:22:00.048 7f22fdda4828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:00.049 7f22fdda4828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:22:00.049 7f22fdda4828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:00.089 7f22fdcd10f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:00.091 7f22fdcd10f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:22:00.101 7f22fdcd10f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:00.146 7f22fddaa630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:00.147 7f22fddaa630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:22:00.147 7f22fddaa630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:00.147 7f23069703c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:22:00.147 7f23069703c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:22:00.147 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:22:00.148 7f23069703c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:22:00.152 7f22fde5fe10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:22:00.160 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:22:00.293 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:22:00.308 7f22fde5fe10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:22:00.318 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:22:00.659 7f23069703c8 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:22:00.826 7f23069703c8 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:22:01.081 7f23069703c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:22:01.082 7f23069703c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:22:01.320 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:01.338 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:22:01.357 7f22fde5fe10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:22:01.393 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue'] 11/07/2024 02:22:01.583 7f23069703c8 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 11/07/2024 02:22:01.583 7f23069703c8 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:22:01.583 7f23069703c8 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:22:01.587 7f23069703c8 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:22:01.587 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:22:01.588 7f23069703c8 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:22:01.588 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:22:01.598 7f23069703c8 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:22:01.598 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:22:01.602 7f23069703c8 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:22:01.602 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:22:01.602 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:22:01.602 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:22:01.602 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:22:01.602 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:22:01.602 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:22:01.603 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:22:01.603 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:22:01.603 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:22:01.603 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:22:01.603 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:22:01.606 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:22:01.610 7f23069703c8 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:22:01.610 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:22:01.611 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:22:01.611 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:22:01.612 7f23069703c8 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:22:01.614 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:22:01.614 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:22:01.616 7f23069703c8 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:22:01.616 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:22:01.617 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:22:01.617 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:22:01.619 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:22:01.619 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:22:01.620 7f23069703c8 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:22:01.620 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:22:01.620 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:22:01.621 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:22:01.621 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:22:01.621 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:22:01.621 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:22:01.621 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:22:01.621 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:22:01.621 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:22:01.621 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:22:01.622 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:22:01.622 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:22:01.622 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:22:01.622 7f23069703c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:22:01.634 7f23069703c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:22:01.716 7f22fdc33780 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:22:01.716 7f22fdc33780 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:22:01.716 7f22fd3e3cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:01.717 7f22fd3e3cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:22:01.717 7f22fd3e3cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:01.740 7f22fd44dd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:01.741 7f22fd44dd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:22:01.741 7f22fd44dd68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:02.397 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:02.410 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:22:02.415 7f22fde5fe10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:22:02.429 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:22:03.432 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:03.848 7f22fd2b0e10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:22:03.849 7f22fd2b0e10 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:22:03.849 7f22fd2b0e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:22:03.850 7f22fd2b0e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:22:03.850 7f22fd2b0e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:22:03.850 7f22fd2b0e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:22:03.850 7f22fd2b0e10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:22:03.850 7f22fd2b0e10 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:22:03.851 7f22fd2b0e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:22:03.851 7f22fd2b0e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:22:03.851 7f22fd2b0e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:22:03.851 7f22fd2b28d0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:22:03.852 7f22fd2b28d0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:22:03.855 7f22fd2b2780 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:22:03.856 7f22fd2b2780 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:22:03.857 7f22feb8aeb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:22:03.857 7f22feb8aeb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:22:03.857 7f22feb8aeb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:22:03.858 7f22feb8aeb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:22:03.858 7f22feb8aeb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:22:03.858 7f22fd2b24e0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:22:03.864 7f22fd2b24e0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:22:04.433 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:05.434 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:06.435 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:07.436 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:08.437 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:09.440 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:10.441 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:11.442 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:12.443 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:13.444 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:14.445 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:15.446 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:16.447 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:17.447 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:18.449 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:19.451 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:20.453 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:21.456 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:22.458 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:23.459 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:24.460 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:25.470 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:26.471 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:27.472 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:28.473 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:29.475 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:30.476 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:31.477 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:32.244 7f23069703c8 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:22:32.332 7f23069703c8 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:22:32.479 7f22fde5fe10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:19.927 7f504658af08 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:23:19.927 7f504658af08 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:23:19.927 7f504658af08 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:23:19.928 7f503dc536d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:23:19.928 7f503dc536d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:23:19.928 7f503dc536d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:23:19.928 7f503dc536d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:23:19.929 7f503dc536d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:19.929 7f503dc536d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:23:19.929 7f503dc536d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:19.929 7f503dc536d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:19.929 7f503dc536d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 105 start_time = 1730946199928736 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:23:19.929 7f503dc536d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:19.967 7f504658af08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:23:19.972 7f504658af08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:23:19.972 7f503dba5240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:19.973 7f503dba5240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:23:19.973 7f503dba5240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:19.973 7f503dba5240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:19.973 7f503dba5240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:23:19.973 7f503dba5240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:19.984 7f504658af08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:23:19.984 7f503dba50f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:19.985 7f503dba50f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:23:19.985 7f503dba50f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:19.992 7f504658af08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:23:19.992 7f503dafb2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:19.993 7f503dafb2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:23:19.993 7f503dafb2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:19.996 7f504658af08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:23:19.997 7f503dafb2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:19.997 7f503dafb2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:23:19.997 7f503dafb2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:20.005 7f504658af08 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:23:20.005 7f504658af08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:23:20.006 7f504658af08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:23:20.006 7f504658af08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:23:20.007 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:23:20.007 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:23:20.007 7f50406680f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:23:20.007 7f50406680f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:23:20.007 7f503dafbcc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:20.008 7f503dafbcc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:23:20.008 7f503dafbcc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:20.009 7f504658af08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:23:20.015 7f5046497588 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:23:20.015 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:23:20.015 7f50406680f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:23:20.037 7f504658af08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:23:20.038 7f503d9db780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:20.039 7f503d9db780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:23:20.039 7f503d9db780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:20.044 7f504658af08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:23:20.044 7f503d9dcd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:20.044 7f503d9dcd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:23:20.044 7f503d9dcd68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:20.048 7f504658af08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:23:20.056 7f504658af08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:23:20.057 7f503da21828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:20.057 7f503da21828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:23:20.057 7f503da21828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:20.079 7f503d9c30f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:20.079 7f503d9c30f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:23:20.079 7f503d9c30f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:20.098 7f503da26630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:20.098 7f503da26630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:23:20.099 7f503da26630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:20.100 7f504658af08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:23:20.101 7f504658af08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:23:20.101 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:23:20.103 7f504658af08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:23:20.107 7f5046497588 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:23:20.126 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:23:20.133 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:23:20.137 7f5046497588 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:23:20.143 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:23:20.616 7f504658af08 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:23:20.795 7f504658af08 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:23:21.011 7f504658af08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:23:21.012 7f504658af08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:23:21.145 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:21.154 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:23:21.158 7f5046497588 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:23:21.169 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue'] 11/07/2024 02:23:21.512 7f504658af08 [contrail-device-manager] [WARNING]: Process 105 prepared to run in Full mode... 11/07/2024 02:23:21.512 7f504658af08 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:23:21.512 7f504658af08 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:23:21.516 7f504658af08 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:23:21.516 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:23:21.517 7f504658af08 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:23:21.517 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:23:21.519 7f504658af08 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:23:21.519 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:23:21.524 7f504658af08 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:23:21.524 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:23:21.524 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:23:21.524 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:23:21.524 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:23:21.524 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:23:21.524 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:23:21.524 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:23:21.524 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:23:21.525 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:23:21.525 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:23:21.525 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:23:21.525 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:23:21.527 7f504658af08 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:23:21.527 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:23:21.527 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:23:21.527 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:23:21.528 7f504658af08 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:23:21.528 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:23:21.528 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:23:21.528 7f504658af08 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:23:21.528 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:23:21.528 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:23:21.528 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:23:21.529 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:23:21.529 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:23:21.530 7f504658af08 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:23:21.530 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:23:21.530 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:23:21.530 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:23:21.530 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:23:21.530 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:23:21.530 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:23:21.530 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:23:21.530 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:23:21.530 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:23:21.531 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:23:21.531 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:23:21.531 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:23:21.531 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:23:21.531 7f504658af08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:23:21.535 7f504658af08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:23:21.590 7f503d8abf60 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:23:21.590 7f503d8abf60 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:23:21.591 7f503d05acc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:21.591 7f503d05acc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:23:21.591 7f503d05acc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:21.607 7f503d0c4d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:21.607 7f503d0c4d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:23:21.607 7f503d0c4d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:23:22.171 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:22.181 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:23:22.192 7f5046497588 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:23:22.217 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:23:23.222 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:23.928 7f503c72e198 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:23:23.933 7f503c72e198 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:23:23.933 7f503c72e198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:23:23.934 7f503c72e198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:23:23.934 7f503c72e198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:23:23.935 7f503c72e198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:23:23.935 7f503c72e198 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:23:23.935 7f503c72e198 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:23:23.935 7f503c72e198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:23:23.935 7f503c72e198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:23:23.935 7f503c72e198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:23:23.936 7f503c722eb8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:23:23.937 7f503c722eb8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:23:23.952 7f503c722f60 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:23:23.952 7f503c722f60 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:23:23.953 7f503c75b438 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:23:23.953 7f503c75b438 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:23:23.953 7f503c75b438 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:23:23.953 7f503c75b438 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:23:23.954 7f503c75b438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:23:23.954 7f503c722240 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:23:23.955 7f503c722240 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:23:24.223 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:25.225 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:26.226 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:27.228 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:28.228 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:29.230 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:30.231 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:31.232 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:32.233 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:33.235 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:34.236 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:35.237 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:36.238 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:37.248 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:38.248 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:39.252 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:40.253 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:41.254 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:42.255 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:43.256 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:44.257 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:45.258 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:46.259 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:47.261 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:48.262 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:49.263 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:50.266 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:51.267 7f5046497588 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:51.669 7f504658af08 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:23:51.753 7f504658af08 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:24:39.895 7fe7d49eab88 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:24:39.895 7fe7d49eab88 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:24:39.895 7fe7d49eab88 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:24:39.895 7fe7cc03f6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:24:39.895 7fe7cc03f6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:24:39.896 7fe7cc03f6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:24:39.896 7fe7cc03f6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:24:39.896 7fe7cc03f6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.897 7fe7cc03f6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:24:39.897 7fe7cc03f6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.897 7fe7cc03f6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.897 7fe7cc03f6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1730946279896539 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:24:39.897 7fe7cc03f6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.934 7fe7d49eab88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:24:39.939 7fe7d49eab88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:24:39.939 7fe7c7f57240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.940 7fe7c7f57240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:24:39.940 7fe7c7f57240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.940 7fe7c7f57240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.940 7fe7c7f57240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:24:39.940 7fe7c7f57240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.945 7fe7d49eab88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:24:39.946 7fe7c7f570f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.946 7fe7c7f570f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:24:39.946 7fe7c7f570f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.953 7fe7d49eab88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:24:39.954 7fe7c7ed52e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.954 7fe7c7ed52e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:24:39.954 7fe7c7ed52e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.958 7fe7d49eab88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:24:39.960 7fe7c7ed52e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.960 7fe7c7ed52e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:24:39.960 7fe7c7ed52e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.968 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:24:39.968 7fe7d49eab88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:24:39.968 7fe7d49eab88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:24:39.968 7fe7d49eab88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:24:39.969 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:24:39.969 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:24:39.969 7fe7c7f51048 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:24:39.970 7fe7c7f51048 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:24:39.970 7fe7c7ed58d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.970 7fe7c7ed58d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:24:39.970 7fe7c7ed58d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.972 7fe7d49eab88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:24:39.981 7fe7d192f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:24:39.981 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:24:39.981 7fe7c7f51048 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:24:39.997 7fe7d49eab88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:24:39.998 7fe7c7db8780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:39.998 7fe7c7db8780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:24:39.999 7fe7c7db8780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:40.005 7fe7d49eab88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:24:40.005 7fe7c7db9d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:40.005 7fe7c7db9d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:24:40.005 7fe7c7db9d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:40.010 7fe7d49eab88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:24:40.030 7fe7d49eab88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:24:40.030 7fe7c7df7828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:40.030 7fe7c7df7828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:24:40.030 7fe7c7df7828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:40.052 7fe7c7e7e828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:40.053 7fe7c7e7e828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:24:40.053 7fe7c7e7e828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:40.065 7fe7c7e63d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:40.065 7fe7c7e63d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:24:40.065 7fe7c7e63d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:40.065 7fe7d49eab88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:24:40.066 7fe7d49eab88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:24:40.066 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:24:40.066 7fe7d49eab88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:24:40.069 7fe7d192f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:24:40.080 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:24:40.092 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:24:40.097 7fe7d192f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:24:40.109 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:24:40.578 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:24:40.748 7fe7d49eab88 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:24:41.113 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:41.300 7fe7d49eab88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:24:41.305 7fe7d49eab88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:24:41.806 7fe7d49eab88 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 11/07/2024 02:24:41.806 7fe7d49eab88 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:24:41.806 7fe7d49eab88 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:24:41.811 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:24:41.811 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:24:41.813 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:24:41.813 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:24:41.814 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:24:41.814 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:24:41.827 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:24:41.827 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:24:41.827 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:24:41.827 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:24:41.827 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:24:41.827 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:24:41.827 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:24:41.828 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:24:41.828 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:24:41.828 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:24:41.828 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:24:41.828 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:24:41.828 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:24:41.831 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:24:41.831 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:24:41.831 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:24:41.831 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:24:41.832 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:24:41.832 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:24:41.832 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:24:41.833 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:24:41.833 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:24:41.834 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:24:41.834 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:24:41.834 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:24:41.834 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:24:41.835 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:24:41.835 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:24:41.836 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:24:41.836 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:24:41.836 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:24:41.836 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:24:41.836 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:24:41.836 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:24:41.836 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:24:41.837 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:24:41.837 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:24:41.837 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:24:41.837 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:24:41.837 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:24:41.837 7fe7d49eab88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:24:41.842 7fe7d49eab88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:24:41.910 7fe7c7cdb630 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:24:41.910 7fe7c7cdb630 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:24:41.911 7fe7c743aa20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:41.911 7fe7c743aa20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:24:41.911 7fe7c743aa20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:41.950 7fe7c749eac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:41.951 7fe7c749eac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:24:41.951 7fe7c749eac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:42.115 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:42.128 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:24:42.133 7fe7d192f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:24:42.182 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:24:43.195 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:43.903 7fe7c7302390 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:24:43.905 7fe7c7302390 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:24:43.905 7fe7c7302390 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:24:43.905 7fe7c7302390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:43.905 7fe7c7302390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:24:43.905 7fe7c7302390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:43.905 7fe7c7302390 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:24:43.905 7fe7c7302390 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:24:43.906 7fe7c7302390 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:24:43.906 7fe7c7302390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:24:43.906 7fe7c7302390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:24:43.906 7fe7c72fe978 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:24:43.907 7fe7c72fe978 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:24:43.910 7fe7c72fe828 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:24:43.910 7fe7c72fe828 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:24:43.910 7fe7ccbedeb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:24:43.911 7fe7ccbedeb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:24:43.911 7fe7ccbedeb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:24:43.911 7fe7ccbedeb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:24:43.911 7fe7ccbedeb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:24:43.911 7fe7c72fe588 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:24:43.912 7fe7c72fe588 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:24:44.196 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:45.198 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:46.199 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:47.200 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:48.201 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:49.205 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:50.207 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:51.209 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:52.211 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:53.213 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:54.214 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:55.216 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:55.887 7fe7c7302630 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:24:56.217 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:57.252 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:58.253 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:59.254 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:00.256 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:00.271 7fe7c7e5f630 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:25:01.258 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:02.259 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:03.259 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:04.261 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:05.261 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:06.262 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:07.264 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:08.266 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:09.267 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:10.269 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:11.270 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:12.272 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:13.279 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:14.280 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:15.287 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:15.973 7fe7d49eab88 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:25:16.057 7fe7d49eab88 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:25:16.379 7fe7d192f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:00.000 7f2133ae0fc8 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:26:00.000 7f2133ae0fc8 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:26:00.000 7f2133ae0fc8 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:26:00.001 7f212b1686d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:26:00.001 7f212b1686d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:26:00.001 7f212b1686d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:26:00.002 7f212b1686d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:26:00.002 7f212b1686d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.002 7f212b1686d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:26:00.002 7f212b1686d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.002 7f212b1686d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.002 7f212b1686d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 105 start_time = 1730946360001747 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:26:00.002 7f212b1686d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.042 7f2133ae0fc8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:26:00.048 7f2133ae0fc8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:26:00.049 7f212b0bb240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.049 7f212b0bb240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:26:00.049 7f212b0bb240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.049 7f212b0bb240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.049 7f212b0bb240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:26:00.049 7f212b0bb240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.055 7f2133ae0fc8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:26:00.063 7f2133ae0fc8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:26:00.064 7f212b0bb0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.064 7f212b0bb0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:26:00.064 7f212b0bb0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.064 7f212b0bb0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.064 7f212b0bb0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:26:00.064 7f212b0bb0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.070 7f2133ae0fc8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:26:00.073 7f212b00e390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.073 7f212b00e390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:26:00.073 7f212b00e390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.823 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:26:00.824 7f2133ae0fc8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:26:00.824 7f2133ae0fc8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:26:00.824 7f2133ae0fc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:26:00.825 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:26:00.825 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:26:00.825 7f212b0b5048 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:26:00.826 7f212b0b5048 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:26:00.826 7f212b00ea20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.827 7f212b00ea20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:26:00.827 7f212b00ea20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.828 7f2133ae0fc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:26:00.836 7f2130a59780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:26:00.837 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:26:00.837 7f212b0b5048 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:26:00.853 7f2133ae0fc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:26:00.855 7f212aef4630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.855 7f212aef4630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:26:00.855 7f212aef4630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.861 7f2133ae0fc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:26:00.861 7f212af496d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.862 7f212af496d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:26:00.862 7f212af496d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.866 7f2133ae0fc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:26:00.875 7f2133ae0fc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:26:00.876 7f212af38198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.876 7f212af38198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:26:00.876 7f212af38198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.898 7f212af38438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.899 7f212af38438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:26:00.899 7f212af38438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.907 7f212aec72e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.907 7f212aec72e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:26:00.907 7f212aec72e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:00.909 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:26:00.909 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:26:00.909 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:26:00.910 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:26:00.914 7f2130a59780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:26:00.921 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:26:00.929 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:26:00.933 7f2130a59780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:26:00.939 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:26:01.422 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:26:01.574 7f2133ae0fc8 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:26:01.826 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:26:01.827 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:26:01.941 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:01.950 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:26:01.954 7f2130a59780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:26:01.967 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue'] 11/07/2024 02:26:02.328 7f2133ae0fc8 [contrail-device-manager] [WARNING]: Process 105 prepared to run in Full mode... 11/07/2024 02:26:02.329 7f2133ae0fc8 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:26:02.329 7f2133ae0fc8 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:26:02.332 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:26:02.333 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:26:02.333 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:26:02.333 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:26:02.335 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:26:02.335 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:26:02.339 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:26:02.339 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:26:02.339 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:26:02.340 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:26:02.340 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:26:02.340 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:26:02.340 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:26:02.340 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:26:02.340 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:26:02.340 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:26:02.340 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:26:02.340 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:26:02.340 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:26:02.342 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:26:02.342 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:26:02.342 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:26:02.342 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:26:02.343 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:26:02.343 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:26:02.343 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:26:02.344 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:26:02.344 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:26:02.344 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:26:02.344 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:26:02.345 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:26:02.345 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:26:02.346 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:26:02.346 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:26:02.346 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:26:02.347 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:26:02.347 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:26:02.347 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:26:02.347 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:26:02.347 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:26:02.347 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:26:02.347 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:26:02.348 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:26:02.348 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:26:02.348 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:26:02.348 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:26:02.348 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:26:02.355 7f2133ae0fc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:26:02.404 7f212ae19cc0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:26:02.404 7f212ae19cc0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:26:02.407 7f212a56ea20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:02.407 7f212a56ea20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:26:02.407 7f212a56ea20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:02.423 7f212a5cfac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:02.424 7f212a5cfac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:26:02.424 7f212a5cfac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:02.971 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:03.200 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:26:03.206 7f2130a59780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:26:03.237 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:26:04.001 7f212a432240 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:04.002 7f212a432240 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:26:04.002 7f212a432240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:04.003 7f212a432240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:04.003 7f212a432240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:04.003 7f212a432240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:04.003 7f212a432240 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:26:04.003 7f212a432240 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:26:04.003 7f212a432240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:26:04.004 7f212a432240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:26:04.004 7f212a432240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:26:04.004 7f212a4406d8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:26:04.005 7f212a4406d8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:26:04.023 7f212a440588 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:26:04.023 7f212a440588 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:26:04.024 7f212bd17eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:26:04.024 7f212bd17eb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:26:04.024 7f212bd17eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:26:04.024 7f212bd17eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:26:04.025 7f212bd17eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:26:04.025 7f212a4402e8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:26:04.026 7f212a4402e8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:26:04.241 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:05.243 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:06.244 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:07.244 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:08.246 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:09.247 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:10.249 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:11.251 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:12.252 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:13.254 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:14.255 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:15.256 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:16.257 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:17.260 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:18.263 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:19.264 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:20.266 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:21.267 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:22.268 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:23.270 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:24.271 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:25.273 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:26.274 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:27.274 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:28.276 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:29.277 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:30.279 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:31.280 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:32.281 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:33.281 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:34.282 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:35.283 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:35.964 7f2133ae0fc8 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:26:36.050 7f2133ae0fc8 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:26:36.446 7f2130a59780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:19.913 7f99ac3228c8 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:27:19.913 7f99ac3228c8 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:27:19.913 7f99ac3228c8 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:27:19.914 7f99a39666d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:27:19.914 7f99a39666d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:27:19.914 7f99a39666d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:27:19.914 7f99a39666d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:27:19.915 7f99a39666d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.915 7f99a39666d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:19.915 7f99a39666d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.915 7f99a39666d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.915 7f99a39666d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1730946439914734 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:27:19.915 7f99a39666d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.950 7f99ac3228c8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:27:19.955 7f99ac3228c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:27:19.956 7f99a38b8240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.956 7f99a38b8240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:27:19.957 7f99a38b8240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.957 7f99a38b8240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.957 7f99a38b8240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:27:19.957 7f99a38b8240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.960 7f99ac3228c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:27:19.961 7f99a38b80f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.961 7f99a38b80f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:19.961 7f99a38b80f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.976 7f99ac3228c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:27:19.977 7f99a38102e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.977 7f99a38102e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:27:19.977 7f99a38102e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.985 7f99ac3228c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:27:19.986 7f99a38102e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.986 7f99a38102e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:19.987 7f99a38102e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.994 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:27:19.994 7f99ac3228c8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:27:19.994 7f99ac3228c8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:27:19.995 7f99ac3228c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:27:19.995 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:27:19.995 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:19.996 7f99a38b2048 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:27:19.996 7f99a38b2048 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:27:19.996 7f99a38108d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.996 7f99a38108d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:27:19.996 7f99a38108d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:19.997 7f99ac3228c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:27:20.024 7f99a9257780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:20.024 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:27:20.025 7f99a38b2048 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:27:20.055 7f99ac3228c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:27:20.056 7f99a36ef780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:20.057 7f99a36ef780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:27:20.057 7f99a36ef780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:20.066 7f99ac3228c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:27:20.066 7f99a36f1d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:20.069 7f99a36f1d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:27:20.069 7f99a36f1d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:20.089 7f99ac3228c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:27:20.099 7f99ac3228c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:27:20.099 7f99a3733828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:20.099 7f99a3733828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:20.099 7f99a3733828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:20.128 7f99a37d1630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:20.129 7f99a37d1630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:27:20.129 7f99a37d1630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:20.136 7f99a36e8e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:20.137 7f99a36e8e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:20.137 7f99a36e8e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:20.137 7f99ac3228c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:27:20.137 7f99ac3228c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:27:20.137 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:20.138 7f99ac3228c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:27:20.152 7f99a9257780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:20.162 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:27:20.171 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:20.214 7f99a9257780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:20.221 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:27:20.649 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:27:20.809 7f99ac3228c8 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:27:21.038 7f99ac3228c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:27:21.038 7f99ac3228c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:27:21.223 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:21.229 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:21.233 7f99a9257780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:21.242 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue'] 11/07/2024 02:27:21.539 7f99ac3228c8 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 11/07/2024 02:27:21.539 7f99ac3228c8 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:27:21.539 7f99ac3228c8 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:27:21.543 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:27:21.543 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:27:21.544 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:27:21.544 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:27:21.545 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:27:21.545 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:27:21.551 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:27:21.551 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:27:21.551 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:27:21.551 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:27:21.552 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:27:21.552 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:27:21.552 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:27:21.552 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:27:21.552 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:27:21.552 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:27:21.557 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:27:21.557 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:27:21.557 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:27:21.559 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:27:21.559 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:27:21.559 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:27:21.559 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:27:21.560 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:27:21.560 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:27:21.560 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:27:21.561 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:27:21.561 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:27:21.561 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:27:21.561 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:27:21.562 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:27:21.562 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:27:21.563 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:27:21.563 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:27:21.563 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:27:21.563 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:27:21.563 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:27:21.564 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:27:21.564 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:27:21.564 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:27:21.564 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:27:21.564 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:27:21.564 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:27:21.564 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:27:21.564 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:27:21.565 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:27:21.565 7f99ac3228c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:27:21.568 7f99ac3228c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:27:21.632 7f99a3616a20 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:27:21.632 7f99a3616a20 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:27:21.632 7f99a2d706d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.632 7f99a2d706d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:27:21.633 7f99a2d706d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.652 7f99a2dcc780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:21.653 7f99a2dcc780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:27:21.653 7f99a2dcc780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:27:22.247 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:22.268 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:27:22.273 7f99a9257780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:27:22.299 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:27:23.303 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:23.915 7f99a2cdad68 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:27:23.927 7f99a2cdad68 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:27:23.927 7f99a2cdad68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:27:23.927 7f99a2cdad68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:27:23.927 7f99a2cdad68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:27:23.927 7f99a2cdad68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:27:23.927 7f99a2cdad68 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:27:23.927 7f99a2cdad68 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:27:23.928 7f99a2cdad68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:27:23.928 7f99a2cdad68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:27:23.928 7f99a2cdad68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:27:23.928 7f99a2c372e8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:27:23.929 7f99a2c372e8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:27:23.935 7f99a2c37198 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:27:23.935 7f99a2c37198 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:27:23.936 7f99a4516eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:27:23.936 7f99a4516eb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:27:23.936 7f99a4516eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:27:23.936 7f99a4516eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:27:23.936 7f99a4516eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:27:23.937 7f99a2ca3ac8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:27:23.938 7f99a2ca3ac8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:27:24.304 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:25.306 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:26.309 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:27.311 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:28.313 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:29.315 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:30.324 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:31.325 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:32.326 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:33.328 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:34.329 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:35.332 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:36.342 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:37.343 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:38.347 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:39.350 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:40.351 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:41.354 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:42.356 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:43.815 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:44.816 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:45.817 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:46.821 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:47.822 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:48.824 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:49.825 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:50.826 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:51.828 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:52.829 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:53.831 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:54.832 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:55.834 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:56.835 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:57.836 7f99a9257780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:58.058 7f99ac3228c8 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:27:58.172 7f99ac3228c8 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:28:39.986 7f06eeadf2c8 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:28:39.986 7f06eeadf2c8 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:28:39.986 7f06eeadf2c8 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:28:39.987 7f06e61786d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:28:39.987 7f06e61786d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:28:39.988 7f06e61786d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:28:39.988 7f06e61786d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:28:39.988 7f06e61786d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:39.988 7f06e61786d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:28:39.988 7f06e61786d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:39.988 7f06e61786d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:39.988 7f06e61786d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1730946519987962 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:28:39.988 7f06e61786d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.027 7f06eeadf2c8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:28:40.032 7f06eeadf2c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:28:40.033 7f06e60c8240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.033 7f06e60c8240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:28:40.033 7f06e60c8240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.033 7f06e60c8240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.033 7f06e60c8240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:28:40.033 7f06e60c8240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.038 7f06eeadf2c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:28:40.038 7f06e60c8240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.039 7f06e60c8240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:28:40.039 7f06e60c8240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.048 7f06eeadf2c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:28:40.048 7f06e601e2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.049 7f06e601e2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:28:40.049 7f06e601e2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.052 7f06eeadf2c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:28:40.053 7f06e601e240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.053 7f06e601e240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:28:40.053 7f06e601e240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.073 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:28:40.074 7f06eeadf2c8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:28:40.074 7f06eeadf2c8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:28:40.074 7f06eeadf2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:28:40.075 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:28:40.075 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:28:40.075 7f06e8b8b0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:28:40.076 7f06e8b8b0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:28:40.076 7f06e601ecc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.076 7f06e601ecc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:28:40.076 7f06e601ecc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.078 7f06eeadf2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:28:40.085 7f06e5ff9e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:28:40.085 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:28:40.088 7f06e8b8b0f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:28:40.119 7f06eeadf2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:28:40.121 7f06e5eff780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.121 7f06e5eff780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:28:40.121 7f06e5eff780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.125 7f06eeadf2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:28:40.126 7f06e5f01d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.126 7f06e5f01d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:28:40.126 7f06e5f01d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.131 7f06eeadf2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:28:40.146 7f06eeadf2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:28:40.147 7f06e5f41828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.147 7f06e5f41828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:28:40.147 7f06e5f41828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.169 7f06e5fdf630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.170 7f06e5fdf630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:28:40.170 7f06e5fdf630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.179 7f06e5ef8e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.179 7f06e5ef8e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:28:40.180 7f06e5ef8e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:40.180 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:28:40.180 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:28:40.180 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:28:40.182 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:28:40.187 7f06e5ff9e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:28:40.202 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:28:40.212 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:28:40.225 7f06e5ff9e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:28:40.244 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:28:40.697 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:28:40.891 7f06eeadf2c8 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:28:41.247 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:42.249 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:42.807 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:28:42.811 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:28:43.250 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:43.308 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:28:43.311 7f06eeadf2c8 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 11/07/2024 02:28:43.311 7f06eeadf2c8 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:28:43.311 7f06eeadf2c8 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:28:43.315 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:28:43.315 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:28:43.316 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:28:43.316 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:28:43.317 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:28:43.317 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:28:43.330 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:28:43.330 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:28:43.330 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:28:43.330 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:28:43.330 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:28:43.330 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:28:43.330 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:28:43.330 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:28:43.330 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:28:43.331 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:28:43.331 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:28:43.331 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:28:43.331 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:28:43.332 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:28:43.332 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:28:43.333 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:28:43.333 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:28:43.333 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:28:43.333 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:28:43.334 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:28:43.334 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:28:43.334 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:28:43.334 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:28:43.334 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:28:43.334 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:28:43.334 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:28:43.335 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:28:43.335 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:28:43.336 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:28:43.336 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:28:43.336 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:28:43.336 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:28:43.336 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:28:43.336 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:28:43.336 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:28:43.336 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:28:43.336 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:28:43.336 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:28:43.336 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:28:43.337 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:28:43.337 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:28:43.340 7f06eeadf2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:28:43.400 7f06e5e254e0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:28:43.400 7f06e5e254e0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:28:43.404 7f06e55e5f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:43.405 7f06e55e5f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:28:43.405 7f06e55e5f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:43.430 7f06e5e38e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:43.430 7f06e5e38e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:28:43.430 7f06e5e38e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:43.438 7f06e5ff9e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:28:43.479 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:28:43.527 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:28:43.531 7f06e5ff9e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:28:43.615 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:28:43.997 7f06e54e82e8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:28:44.002 7f06e54e82e8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:28:44.002 7f06e54e82e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:28:44.002 7f06e54e82e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:44.002 7f06e54e82e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:28:44.002 7f06e54e82e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:44.002 7f06e54e82e8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:28:44.003 7f06e54e82e8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:28:44.003 7f06e54e82e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:28:44.003 7f06e54e82e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:28:44.003 7f06e54e82e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:28:44.003 7f06e54baf60 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:28:44.012 7f06e54baf60 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:28:44.053 7f06e54bac18 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:28:44.054 7f06e54bac18 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:28:44.054 7f06e543fa20 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:28:44.055 7f06e543fa20 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:28:44.055 7f06e543fa20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:28:44.055 7f06e543fa20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:28:44.056 7f06e543fa20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:28:44.056 7f06e54bacc0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:28:44.059 7f06e54bacc0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:28:44.628 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:45.631 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:46.632 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:47.642 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:48.644 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:49.644 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:50.658 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:51.659 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:52.660 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:53.661 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:54.662 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:55.670 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:56.677 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:57.678 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:58.681 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:59.683 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:00.683 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:01.684 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:02.685 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:03.686 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:04.687 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:05.688 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:06.689 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:07.691 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:08.696 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:09.698 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:10.699 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:11.700 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:12.701 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:13.702 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:14.704 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:15.705 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:16.706 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:17.707 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:18.713 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:19.719 7f06e5ff9e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:20.225 7f06eeadf2c8 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:29:20.305 7f06eeadf2c8 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:30:00.045 7f2ceba6b348 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:30:00.045 7f2ceba6b348 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:30:00.045 7f2ceba6b348 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:30:00.045 7f2ce313b6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:30:00.046 7f2ce313b6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:30:00.046 7f2ce313b6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:30:00.047 7f2ce313b6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:30:00.047 7f2ce313b6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.047 7f2ce313b6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:30:00.047 7f2ce313b6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.047 7f2ce313b6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.047 7f2ce313b6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1730946600046759 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:30:00.047 7f2ce313b6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.120 7f2ceba6b348 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:30:00.133 7f2ceba6b348 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:30:00.134 7f2ce308c240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.134 7f2ce308c240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:30:00.134 7f2ce308c240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.134 7f2ce308c240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.135 7f2ce308c240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:30:00.135 7f2ce308c240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.138 7f2ceba6b348 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:30:00.138 7f2ce308c0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.138 7f2ce308c0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:30:00.138 7f2ce308c0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.151 7f2ceba6b348 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:30:00.151 7f2ce2fe42e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.155 7f2ce2fe42e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:30:00.155 7f2ce2fe42e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.161 7f2ceba6b348 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:30:00.162 7f2ce2fe42e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.162 7f2ce2fe42e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:30:00.162 7f2ce2fe42e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.173 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:30:00.173 7f2ceba6b348 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:30:00.174 7f2ceba6b348 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:30:00.174 7f2ceba6b348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:30:00.174 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:30:00.176 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:30:00.177 7f2ce5b510f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:30:00.177 7f2ce5b510f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:30:00.177 7f2ce2fe4cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.178 7f2ce2fe4cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:30:00.178 7f2ce2fe4cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.179 7f2ceba6b348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:30:00.187 7f2ce5b510f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:30:00.188 7f2ceb70dc18 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:30:00.188 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:30:00.224 7f2ceba6b348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:30:00.227 7f2ce2ec7780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.228 7f2ce2ec7780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:30:00.228 7f2ce2ec7780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.235 7f2ceba6b348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:30:00.235 7f2ce2ec9d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.236 7f2ce2ec9d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:30:00.236 7f2ce2ec9d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.242 7f2ceba6b348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:30:00.253 7f2ceba6b348 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:30:00.253 7f2ce2f0a828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.253 7f2ce2f0a828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:30:00.253 7f2ce2f0a828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.278 7f2ce2f0aac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.278 7f2ce2f0aac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:30:00.279 7f2ce2f0aac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.291 7f2ce2e99978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.291 7f2ce2e99978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:30:00.291 7f2ce2e99978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:00.293 7f2ceba6b348 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:30:00.293 7f2ceba6b348 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:30:00.293 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:30:00.294 7f2ceba6b348 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:30:00.302 7f2ceb70dc18 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:30:00.314 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:30:00.325 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:30:00.338 7f2ceb70dc18 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:30:00.357 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:30:00.807 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:30:01.364 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:01.389 7f2ceba6b348 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:30:01.668 7f2ceba6b348 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:30:01.668 7f2ceba6b348 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:30:02.169 7f2ceba6b348 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 11/07/2024 02:30:02.169 7f2ceba6b348 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:30:02.169 7f2ceba6b348 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:30:02.174 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:30:02.174 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:30:02.175 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:30:02.175 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:30:02.176 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:30:02.177 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:30:02.182 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:30:02.182 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:30:02.183 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:30:02.183 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:30:02.183 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:30:02.183 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:30:02.183 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:30:02.184 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:30:02.184 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:30:02.184 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:30:02.184 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:30:02.184 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:30:02.184 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:30:02.187 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:30:02.187 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:30:02.187 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:30:02.187 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:30:02.188 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:30:02.188 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:30:02.188 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:30:02.188 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:30:02.189 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:30:02.189 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:30:02.189 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:30:02.189 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:30:02.189 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:30:02.190 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:30:02.190 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:30:02.190 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:30:02.190 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:30:02.190 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:30:02.190 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:30:02.190 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:30:02.190 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:30:02.190 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:30:02.191 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:30:02.191 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:30:02.191 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:30:02.191 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:30:02.191 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:30:02.191 7f2ceba6b348 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:30:02.195 7f2ceba6b348 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:30:02.282 7f2ce2d27630 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:30:02.282 7f2ce2d27630 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:30:02.282 7f2ce2544828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:02.282 7f2ce2544828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:30:02.282 7f2ce2544828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:02.315 7f2ce25a08d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:02.316 7f2ce25a08d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:30:02.316 7f2ce25a08d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:30:02.366 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:02.382 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:30:02.395 7f2ceb70dc18 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:30:02.440 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:30:03.444 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:04.047 7f2ce2407d68 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:30:04.048 7f2ce2407d68 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:30:04.048 7f2ce2407d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:30:04.048 7f2ce2407d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:30:04.048 7f2ce2407d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:30:04.048 7f2ce2407d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:30:04.048 7f2ce2407d68 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:30:04.048 7f2ce2407d68 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:30:04.049 7f2ce2407d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:30:04.049 7f2ce2407d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:30:04.049 7f2ce2407d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:30:04.049 7f2ce2407b70 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:30:04.050 7f2ce2407b70 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:30:04.055 7f2ce2407240 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:30:04.055 7f2ce2407240 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:30:04.056 7f2ce2407ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:30:04.056 7f2ce2407ac8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:30:04.056 7f2ce2407ac8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:30:04.056 7f2ce2407ac8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:30:04.056 7f2ce2407ac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:30:04.056 7f2ce2407a20 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:30:04.059 7f2ce2407a20 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:30:04.446 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:05.447 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:06.576 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:07.577 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:08.578 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:09.579 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:10.580 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:11.581 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:12.582 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:13.583 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:14.584 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:15.585 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:16.586 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:17.587 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:18.588 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:19.589 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:20.590 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:21.591 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:22.595 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:23.596 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:24.598 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:25.601 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:26.602 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:27.603 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:28.604 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:28.995 7f2ce240ba20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:30:29.605 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:30.607 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:31.609 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:32.610 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:33.612 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:34.614 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:35.615 7f2ceb70dc18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:35.748 7f2ceba6b348 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:30:35.851 7f2ceba6b348 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:31:19.868 7f5cb478d2c8 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:31:19.868 7f5cb478d2c8 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:31:19.868 7f5cb478d2c8 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:31:19.868 7f5cac0666d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:31:19.868 7f5cac0666d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:31:19.869 7f5cac0666d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:31:19.869 7f5cac0666d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:31:19.869 7f5cac0666d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.869 7f5cac0666d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:19.869 7f5cac0666d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.869 7f5cac0666d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.870 7f5cac0666d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1730946679869289 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:31:19.870 7f5cac0666d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.910 7f5cb478d2c8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:31:19.915 7f5cb478d2c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:31:19.916 7f5ca7d84240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.916 7f5ca7d84240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:31:19.916 7f5ca7d84240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.917 7f5ca7d84240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.917 7f5ca7d84240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:31:19.917 7f5ca7d84240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.922 7f5cb478d2c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:31:19.924 7f5ca7d840f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.924 7f5ca7d840f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:19.924 7f5ca7d840f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.931 7f5cb478d2c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:31:19.931 7f5ca7cdd2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.932 7f5ca7cdd2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:31:19.932 7f5ca7cdd2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.936 7f5cb478d2c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:31:19.937 7f5ca7cdd2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.939 7f5ca7cdd2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:19.939 7f5ca7cdd2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.944 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:31:19.944 7f5cb478d2c8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:31:19.944 7f5cb478d2c8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:31:19.944 7f5cb478d2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:31:19.945 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:31:19.945 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:31:19.945 7f5cb1744780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:31:19.946 7f5cb1744780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:31:19.946 7f5ca7cdd2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.946 7f5ca7cdd2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:31:19.946 7f5ca7cdd2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.949 7f5cb478d2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:31:19.966 7f5cb4699c18 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:31:19.967 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:31:19.978 7f5cb1744780 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:31:19.983 7f5cb478d2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:31:19.986 7f5ca7bbc780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:19.986 7f5ca7bbc780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:31:19.987 7f5ca7bbc780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:20.967 7f5cb478d2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:31:20.967 7f5ca7bbfd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:20.968 7f5ca7bbfd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:31:20.968 7f5ca7bbfd68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:20.976 7f5cb478d2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:31:20.986 7f5cb478d2c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:31:20.987 7f5ca7bfe828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:20.988 7f5ca7bfe828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:20.988 7f5ca7bfe828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:21.009 7f5ca7c82cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:21.010 7f5ca7c82cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:31:21.010 7f5ca7c82cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:21.017 7f5ca7cb66d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:21.017 7f5ca7cb66d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:21.018 7f5ca7cb66d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:21.018 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:31:21.018 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:31:21.019 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:31:21.021 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:31:21.026 7f5cb4699c18 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:31:21.043 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:31:21.051 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:31:21.060 7f5cb4699c18 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:31:21.068 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:31:21.532 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:31:21.735 7f5cb478d2c8 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:31:21.964 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:31:21.968 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:31:22.070 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:22.080 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:31:22.089 7f5cb4699c18 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:31:22.100 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue'] 11/07/2024 02:31:22.469 7f5cb478d2c8 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 11/07/2024 02:31:22.469 7f5cb478d2c8 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:31:22.469 7f5cb478d2c8 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:31:22.472 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:31:22.472 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:31:22.473 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:31:22.473 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:31:22.476 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:31:22.476 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:31:22.480 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:31:22.480 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:31:22.480 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:31:22.481 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:31:22.481 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:31:22.481 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:31:22.481 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:31:22.481 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:31:22.481 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:31:22.481 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:31:22.481 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:31:22.481 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:31:22.481 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:31:22.483 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:31:22.483 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:31:22.483 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:31:22.483 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:31:22.484 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:31:22.484 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:31:22.484 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:31:22.485 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:31:22.485 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:31:22.485 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:31:22.485 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:31:22.485 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:31:22.485 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:31:22.486 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:31:22.486 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:31:22.486 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:31:22.486 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:31:22.486 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:31:22.486 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:31:22.487 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:31:22.487 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:31:22.487 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:31:22.487 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:31:22.487 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:31:22.487 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:31:22.487 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:31:22.488 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:31:22.488 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:31:22.491 7f5cb478d2c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:31:22.546 7f5ca7a74390 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:31:22.546 7f5ca7a74390 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:31:22.547 7f5ca723d6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:22.547 7f5ca723d6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:31:22.547 7f5ca723d6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:22.569 7f5ca729a780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:22.570 7f5ca729a780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:22.570 7f5ca729a780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:23.104 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:23.121 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:31:23.141 7f5cb4699c18 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:31:23.183 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:31:23.875 7f5ca7171d68 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:31:23.879 7f5ca7171d68 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:31:23.879 7f5ca7171d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:31:23.879 7f5ca7171d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:31:23.879 7f5ca7171d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:31:23.879 7f5ca7171d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:31:23.879 7f5ca7171d68 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:31:23.879 7f5ca7171d68 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:31:23.880 7f5ca7171d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:31:23.880 7f5ca7171d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:31:23.880 7f5ca7171d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:31:23.881 7f5ca7106630 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:31:23.886 7f5ca7106630 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:31:23.897 7f5ca71064e0 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:31:23.898 7f5ca71064e0 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:31:23.898 7f5caca01eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:31:23.898 7f5caca01eb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:31:23.899 7f5caca01eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:31:23.899 7f5caca01eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:31:23.899 7f5caca01eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:31:23.900 7f5ca7106240 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:31:23.901 7f5ca7106240 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:31:24.188 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:25.189 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:26.190 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:27.192 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:28.193 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:29.193 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:30.196 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:31.198 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:32.199 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:33.201 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:34.203 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:35.205 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:36.207 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:37.208 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:38.210 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:39.211 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:40.212 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:41.213 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:42.214 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:43.215 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:43.742 7f5caca70d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:31:44.216 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:45.217 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:46.219 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:47.221 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:48.222 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:49.223 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:50.224 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:51.225 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:52.225 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:53.226 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:54.227 7f5cb4699c18 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:54.233 7f5cb478d2c8 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 034cd2c2-70ad-4082-8c89-976eab8c8655 11/07/2024 02:31:54.318 7f5cb478d2c8 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: a1ffa585-5722-4493-a52a-4785657627ca 11/07/2024 02:32:39.943 7f635f80fe48 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:32:39.943 7f635f80fe48 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:32:39.943 7f635f80fe48 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:32:39.943 7f6356efa6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:32:39.944 7f6356efa6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:32:39.944 7f6356efa6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:32:39.945 7f6356efa6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:32:39.945 7f6356efa6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.945 7f6356efa6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:32:39.945 7f6356efa6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.945 7f6356efa6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.945 7f6356efa6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 106 start_time = 1730946759944696 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:32:39.945 7f6356efa6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.988 7f635f80fe48 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//'] 11/07/2024 02:32:39.994 7f635f80fe48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:32:39.994 7f6356e4a240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.995 7f6356e4a240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:32:39.995 7f6356e4a240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.995 7f6356e4a240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:39.995 7f6356e4a240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:32:39.995 7f6356e4a240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.000 7f635f80fe48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:32:40.001 7f6356e4a0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.001 7f6356e4a0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:32:40.001 7f6356e4a0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.013 7f635f80fe48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:32:40.014 7f6356da12e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.014 7f6356da12e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:32:40.014 7f6356da12e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.018 7f635f80fe48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:32:40.019 7f6356da1240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.019 7f6356da1240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:32:40.019 7f6356da1240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.028 7f635f80fe48 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:32:40.029 7f635f80fe48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:32:40.029 7f635f80fe48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:32:40.029 7f635f80fe48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:32:40.030 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:32:40.030 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:32:40.031 7f635990d0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:32:40.031 7f635990d0f0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:32:40.032 7f6356da1cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.032 7f6356da1cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:32:40.032 7f6356da1cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.033 7f635f80fe48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:32:40.040 7f635c7e9780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:32:40.040 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:32:40.040 7f635990d0f0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:32:40.063 7f635f80fe48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:32:40.065 7f6356c81780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.066 7f6356c81780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:32:40.066 7f6356c81780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.078 7f635f80fe48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:32:40.078 7f6356c83d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.079 7f6356c83d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:32:40.079 7f6356c83d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.089 7f635f80fe48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:32:40.098 7f635f80fe48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:32:40.099 7f6356cc3828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.099 7f6356cc3828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:32:40.099 7f6356cc3828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.118 7f635b464390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.118 7f635b464390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:32:40.118 7f635b464390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.128 7f6356cc1630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.128 7f6356cc1630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:32:40.128 7f6356cc1630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:40.129 7f635f80fe48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:32:40.129 7f635f80fe48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:32:40.129 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:32:40.131 7f635f80fe48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1 11/07/2024 02:32:40.135 7f635c7e9780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:32:40.141 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:32:40.147 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:32:40.151 7f635c7e9780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:32:40.158 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1'] 11/07/2024 02:32:40.643 7f635f80fe48 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:32:40.798 7f635f80fe48 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:32:41.027 7f635f80fe48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue 11/07/2024 02:32:41.029 7f635f80fe48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue 11/07/2024 02:32:41.168 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:41.176 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:32:41.180 7f635c7e9780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:32:41.190 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue'] 11/07/2024 02:32:41.530 7f635f80fe48 [contrail-device-manager] [WARNING]: Process 106 prepared to run in Full mode... 11/07/2024 02:32:41.530 7f635f80fe48 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:32:41.530 7f635f80fe48 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:32:41.534 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:32:41.534 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:32:41.535 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:32:41.535 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:32:41.535 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:32:41.536 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:32:41.540 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:32:41.540 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:32:41.540 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:32:41.540 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:32:41.540 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:32:41.540 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:32:41.540 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:32:41.540 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:32:41.540 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:32:41.540 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:32:41.540 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:32:41.541 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:32:41.541 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:32:41.542 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:32:41.542 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:32:41.542 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:32:41.542 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:32:41.543 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:32:41.543 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:32:41.543 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:32:41.543 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:32:41.544 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:32:41.544 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:32:41.544 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:32:41.544 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:32:41.544 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:32:41.545 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:32:41.545 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:32:41.545 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:32:41.545 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:32:41.545 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:32:41.545 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:32:41.545 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:32:41.545 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:32:41.545 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:32:41.545 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:32:41.545 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:32:41.546 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:32:41.546 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:32:41.546 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:32:41.546 7f635f80fe48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:32:41.549 7f635f80fe48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:32:41.597 7f6356b31588 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:32:41.597 7f6356b31588 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:32:41.597 7f63563006d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:41.598 7f63563006d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:32:41.598 7f63563006d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:41.616 7f635635d780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:41.617 7f635635d780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:32:41.617 7f635635d780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:32:42.193 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:42.201 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:32:42.205 7f635c7e9780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:32:42.217 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-k8s-764-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-k8s-764-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:32:43.220 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:43.944 7f6356234d68 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:32:43.946 7f6356234d68 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:32:43.946 7f6356234d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:32:43.946 7f6356234d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:32:43.946 7f6356234d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-764-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:32:43.946 7f6356234d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:32:43.946 7f6356234d68 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:32:43.946 7f6356234d68 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:32:43.947 7f6356234d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:32:43.947 7f6356234d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:32:43.947 7f6356234d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:32:43.947 7f63561c8630 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:32:43.949 7f63561c8630 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:32:43.952 7f63561c84e0 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001013 11/07/2024 02:32:43.953 7f63561c84e0 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:32:43.953 7f6357aa6eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:32:43.953 7f6357aa6eb8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:32:43.953 7f6357aa6eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:32:43.954 7f6357aa6eb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:32:43.954 7f6357aa6eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:32:43.954 7f63561c8240 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:32:43.955 7f63561c8240 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:32:44.221 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:45.222 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:46.224 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:47.225 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:48.226 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:49.233 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:50.235 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:51.236 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:52.236 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:53.239 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:54.241 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:55.242 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:56.243 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:57.246 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:58.247 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:59.248 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:00.249 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:01.251 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:02.252 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:03.253 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:04.254 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:05.265 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:05.340 7f635623c4e0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:33:06.266 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:07.267 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:08.267 7f635c7e9780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process.