11/07/2024 02:09:33.095 7fbed87a6e88 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:09:33.095 7fbed87a6e88 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:09:33.096 7fbed87a6e88 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:09:33.096 7fbed00a36d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:09:33.097 7fbed00a36d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:09:33.098 7fbed00a36d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:09:33.098 7fbed00a36d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:09:33.098 7fbed00a36d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:33.099 7fbed00a36d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:09:33.099 7fbed00a36d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:33.099 7fbed00a36d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:33.099 7fbed00a36d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 120 start_time = 1730945373097995 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:09:33.100 7fbed00a36d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:33.216 7fbed87a6e88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:09:33.239 7fbed87a6e88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:09:33.240 7fbed0073978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:33.240 7fbed0073978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:09:33.240 7fbed0073978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:33.240 7fbed0073978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:33.240 7fbed0073978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:09:33.240 7fbed0073978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:37.098 7fbed5781780 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:09:42.102 7fbed5781780 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 11/07/2024 02:09:42.102 7fbed5781780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:09:42.103 7fbed5781780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:09:42.103 7fbed5781780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:09:42.103 7fbed5781780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:09:42.103 7fbed5781780 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/07/2024 02:09:42.103 7fbed5781780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/07/2024 02:09:42.104 7fbed5781780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:42.104 7fbed5781780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:09:42.104 7fbed5781780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:42.104 7fbed5781780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:42.104 7fbed5781780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 120 start_time = 1730945373097995 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/07/2024 02:09:42.104 7fbed5781780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:09:46.103 7fbed0073780 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:09:46.111 7fbed0073780 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:09:46.111 7fbed0073780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:09:46.111 7fbed0073780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:09:46.111 7fbed0073780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:09:46.111 7fbed0073780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:09:46.111 7fbed0073780 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:09:46.112 7fbed0073780 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:09:46.112 7fbed0073780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:09:46.112 7fbed0073780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:09:46.112 7fbed0073780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:09:46.112 7fbed004e780 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:09:46.113 7fbed004e780 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:09:46.125 7fbed004e630 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:09:46.125 7fbed004e630 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:09:46.126 7fbed002e390 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:09:46.127 7fbed002e390 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:09:46.128 7fbed002e390 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:09:46.128 7fbed002e390 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:09:46.129 7fbed002e390 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:09:46.129 7fbed004e2e8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:09:46.130 7fbed004e2e8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:09:53.287 7fbed004e630 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001289 11/07/2024 02:09:53.287 7fbed004e630 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:09:53.288 7fbed87a6e88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:09:53.289 7fbed003b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 11/07/2024 02:09:53.289 7fbed003b8d0 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 2 11/07/2024 02:09:53.289 7fbed003b8d0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:09:53.289 7fbed003b8d0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:09:53.290 7fbed0a44e10 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:09:53.291 7fbed0a44e10 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:09:53.352 7fbed87a6e88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:09:53.380 7fbed87a6e88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:09:53.396 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:09:53.398 7fbed87a6e88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:09:53.398 7fbed87a6e88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:09:53.398 7fbed87a6e88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:09:53.399 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:09:53.399 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:09:53.399 7fbed0073438 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:09:53.399 7fbed0073438 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:09:53.403 7fbed87a6e88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:09:53.456 7fbed0073438 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:09:53.458 7fbed00738d0 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:09:53.458 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:09:53.542 7fbed87a6e88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:09:53.562 7fbed87a6e88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:09:53.589 7fbed87a6e88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:09:53.713 7fbed87a6e88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:09:53.718 7fbed87a6e88 [contrail-device-manager] [ERROR]: Error in db walk read 'type' 11/07/2024 02:09:53.731 7fbed87a6e88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:09:53.731 7fbed87a6e88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:09:53.732 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:09:53.732 7fbed87a6e88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:09:53.751 7fbed00738d0 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:09:53.773 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:09:53.782 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:09:53.802 7fbed00738d0 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:09:53.813 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:09:54.259 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:09:54.481 7fbed87a6e88 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:09:54.732 7fbed87a6e88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:09:54.732 7fbed87a6e88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:09:54.815 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:54.852 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:09:54.871 7fbed00738d0 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:09:54.887 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:09:55.234 7fbed87a6e88 [contrail-device-manager] [WARNING]: Process 120 prepared to run in Full mode... 11/07/2024 02:09:55.234 7fbed87a6e88 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:09:55.234 7fbed87a6e88 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:09:55.242 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:09:55.242 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:09:55.244 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:09:55.245 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:09:55.246 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:09:55.247 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:09:55.262 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:09:55.262 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:09:55.262 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:09:55.262 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:09:55.263 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:09:55.263 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:09:55.263 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:09:55.263 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:09:55.263 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:09:55.263 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:09:55.263 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:09:55.263 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:09:55.263 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:09:55.269 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:09:55.270 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:09:55.270 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:09:55.270 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:09:55.271 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:09:55.271 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:09:55.271 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:09:55.272 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:09:55.272 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:09:55.272 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:09:55.272 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:09:55.273 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:09:55.273 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:09:55.275 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:09:55.275 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:09:55.276 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:09:55.276 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:09:55.276 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:09:55.276 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:09:55.276 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:09:55.276 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:09:55.276 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:09:55.276 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:09:55.276 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:09:55.277 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:09:55.277 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:09:55.277 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:09:55.277 7fbed87a6e88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:09:55.283 7fbed87a6e88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:09:55.537 7fbecaadaeb8 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:09:55.538 7fbecaadaeb8 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:09:55.890 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:55.910 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:09:55.928 7fbed00738d0 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:09:55.988 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:09:56.684 7fbecabf7198 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:09:56.999 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:58.001 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:09:59.002 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:00.004 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:01.005 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:02.007 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:03.008 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:04.010 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:05.011 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:06.012 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:07.017 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:08.018 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:09.019 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:10.027 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:11.331 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:12.332 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:13.337 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:14.340 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:15.345 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:16.346 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:16.864 7fbeca0a8ac8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:10:17.348 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:18.351 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:19.354 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:20.355 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:21.357 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:22.358 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:23.359 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:24.361 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:25.363 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:26.364 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:27.365 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:28.367 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:29.369 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:30.372 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:31.372 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:32.374 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:33.377 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:34.377 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:35.378 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:36.379 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:37.382 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:38.383 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:39.384 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:40.384 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:41.385 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:42.387 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:43.388 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:44.389 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:45.390 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:45.995 7fbeca1a3b70 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:10:46.391 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:47.392 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:48.395 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:49.396 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:50.396 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:51.289 7fbeca1a3ac8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:10:51.397 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:52.399 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:53.399 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:54.401 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:55.402 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:56.403 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:56.429 7fbed0a9b630 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:10:57.404 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:58.405 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:10:59.406 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:00.407 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:01.409 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:02.410 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:02.519 7fbeca1a3c18 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:11:03.410 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:04.412 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:05.413 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:06.393 7fbed5937e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:11:06.414 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:07.416 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:08.417 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:09.421 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:10.422 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:11.378 7fbed5937e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:11:11.423 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:12.424 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:13.426 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:14.426 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:15.427 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:16.429 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:17.429 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:18.431 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:19.432 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:20.433 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:21.434 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:22.436 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:23.437 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:24.438 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:25.440 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:11:26.455 7fbed00738d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:10.083 7f713bc118c8 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:12:10.084 7f713bc118c8 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:12:10.084 7f713bc118c8 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:12:10.084 7f71332656d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:12:10.084 7f71332656d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:12:10.085 7f71332656d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:12:10.085 7f71332656d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:12:10.085 7f71332656d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.085 7f71332656d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:12:10.085 7f71332656d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.086 7f71332656d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.086 7f71332656d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 111 start_time = 1730945530085374 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:12:10.086 7f71332656d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.140 7f713bc118c8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:12:10.153 7f713bc118c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:12:10.154 7f71331b86d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.154 7f71331b86d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:12:10.154 7f71331b86d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.154 7f71331b86d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.154 7f71331b86d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:12:10.155 7f71331b86d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.169 7f713bc118c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:12:10.170 7f71331b1048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.170 7f71331b1048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:12:10.170 7f71331b1048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.181 7f713bc118c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:12:10.181 7f71330f32e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.182 7f71330f32e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:12:10.182 7f71330f32e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.196 7f713bc118c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:12:10.197 7f71330f3978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.197 7f71330f3978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:12:10.197 7f71330f3978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.207 7f713bc118c8 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:12:10.208 7f713bc118c8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:12:10.208 7f713bc118c8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:12:10.209 7f713bc118c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:12:10.209 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:12:10.209 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:12:10.210 7f71330f2a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:12:10.210 7f71330f2a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:12:10.210 7f71330f3d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.211 7f71330f3d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:12:10.211 7f71330f3d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.212 7f713bc118c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:12:10.232 7f7138b55780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:12:10.233 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:12:10.233 7f71330f2a20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:12:10.285 7f713bc118c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:12:10.287 7f71327f2b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.287 7f71327f2b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:12:10.288 7f71327f2b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.294 7f713bc118c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:12:10.295 7f713287dc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.295 7f713287dc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:12:10.295 7f713287dc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.312 7f713bc118c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:12:10.362 7f713bc118c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:12:10.362 7f7132832390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.363 7f7132832390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:12:10.363 7f7132832390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.377 7f7132832b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.377 7f7132832b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:12:10.378 7f7132832b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.388 7f713283fb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.388 7f713283fb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:12:10.388 7f713283fb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:10.388 7f713bc118c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:12:10.388 7f713bc118c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:12:10.389 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:12:10.389 7f713bc118c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:12:10.402 7f7138b55780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:12:10.412 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:12:10.419 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:12:10.433 7f7138b55780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:12:10.441 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:12:10.911 7f713bc118c8 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:12:11.108 7f713bc118c8 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:12:11.292 7f713bc118c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:12:11.293 7f713bc118c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:12:11.444 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:11.452 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:12:11.468 7f7138b55780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:12:11.482 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:12:11.794 7f713bc118c8 [contrail-device-manager] [WARNING]: Process 111 prepared to run in Full mode... 11/07/2024 02:12:11.794 7f713bc118c8 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:12:11.794 7f713bc118c8 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:12:11.801 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:12:11.801 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:12:11.804 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:12:11.804 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:12:11.807 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:12:11.807 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:12:11.817 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:12:11.817 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:12:11.817 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:12:11.817 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:12:11.817 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:12:11.817 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:12:11.817 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:12:11.818 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:12:11.818 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:12:11.818 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:12:11.818 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:12:11.818 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:12:11.818 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:12:11.820 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:12:11.820 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:12:11.820 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:12:11.820 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:12:11.821 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:12:11.821 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:12:11.821 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:12:11.822 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:12:11.822 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:12:11.822 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:12:11.822 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:12:11.822 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:12:11.822 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:12:11.823 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:12:11.823 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:12:11.823 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:12:11.823 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:12:11.823 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:12:11.823 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:12:11.823 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:12:11.824 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:12:11.824 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:12:11.824 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:12:11.824 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:12:11.824 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:12:11.824 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:12:11.824 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:12:11.824 7f713bc118c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:12:11.827 7f713bc118c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:12:11.912 7f71325fe0f0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:12:11.912 7f71325fe0f0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:12:11.913 7f7131ee6438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:11.913 7f7131ee6438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:12:11.914 7f7131ee6438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:11.953 7f7131eb44e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:11.953 7f7131eb44e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:12:11.953 7f7131eb44e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:12:12.486 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:12.499 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:12:12.515 7f7138b55780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:12:12.533 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:12:13.536 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:14.085 7f7131d90c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:12:14.093 7f7131d90c18 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:12:14.093 7f7131d90c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:12:14.093 7f7131d90c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:12:14.093 7f7131d90c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:12:14.093 7f7131d90c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:12:14.093 7f7131d90c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:12:14.094 7f7131d90c18 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:12:14.094 7f7131d90c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:12:14.095 7f7131d90c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:12:14.095 7f7131d90c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:12:14.095 7f7131da6438 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:12:14.096 7f7131da6438 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:12:14.099 7f7131da62e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:12:14.099 7f7131da62e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:12:14.099 7f7133e18e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:12:14.100 7f7133e18e10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:12:14.100 7f7133e18e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:12:14.100 7f7133e18e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:12:14.100 7f7133e18e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:12:14.101 7f7131da6048 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:12:14.102 7f7131da6048 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:12:14.537 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:15.539 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:16.540 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:17.541 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:18.543 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:19.544 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:20.545 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:21.546 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:22.547 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:23.548 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:24.549 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:25.550 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:26.551 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:27.552 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:28.554 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:29.556 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:30.556 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:31.557 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:32.558 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:33.560 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:34.561 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:35.562 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:36.564 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:37.565 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:38.566 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:39.567 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:40.568 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:41.569 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:42.570 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:43.570 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:44.571 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:45.573 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:46.574 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:47.575 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:48.575 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:49.576 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:50.577 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:51.578 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:52.579 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:53.581 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:54.582 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:55.583 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:56.584 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:57.586 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:58.588 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:12:59.589 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:00.590 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:01.592 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:02.593 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:03.594 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:04.595 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:05.597 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:06.597 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:07.599 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:08.600 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:09.601 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:10.604 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:11.604 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:12.606 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:13.367 7f713bc118c8 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:13:13.527 7f713bc118c8 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:13:13.607 7f7138b55780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:56.096 7fefda8f2708 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:13:56.096 7fefda8f2708 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:13:56.096 7fefda8f2708 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:13:56.096 7fefd1f666d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:13:56.096 7fefd1f666d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:13:56.097 7fefd1f666d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:13:56.097 7fefd1f666d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:13:56.097 7fefd1f666d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.097 7fefd1f666d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:13:56.097 7fefd1f666d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.098 7fefd1f666d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.098 7fefd1f666d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730945636097140 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:13:56.098 7fefd1f666d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.135 7fefda8f2708 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:13:56.140 7fefda8f2708 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:13:56.140 7fefd1eb76d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.141 7fefd1eb76d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:13:56.141 7fefd1eb76d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.141 7fefd1eb76d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.141 7fefd1eb76d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:13:56.141 7fefd1eb76d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.156 7fefda8f2708 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:13:56.157 7fefe07ba630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.158 7fefe07ba630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:13:56.158 7fefe07ba630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.166 7fefda8f2708 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:13:56.167 7fefd1df12e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.167 7fefd1df12e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:13:56.167 7fefd1df12e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.183 7fefda8f2708 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:13:56.194 7fefd1df1978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.195 7fefd1df1978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:13:56.195 7fefd1df1978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.204 7fefda8f2708 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:13:56.205 7fefda8f2708 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:13:56.205 7fefda8f2708 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:13:56.206 7fefda8f2708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:13:56.206 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:13:56.206 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:13:56.206 7fefd1df0f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:13:56.207 7fefd1df0f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:13:56.207 7fefd1df1d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.207 7fefd1df1d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:13:56.207 7fefd1df1d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.209 7fefda8f2708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:13:56.225 7fefd1df0f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:13:56.226 7fefd1df0a20 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:13:56.226 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:13:56.281 7fefda8f2708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:13:56.283 7fefd14f2b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.284 7fefd14f2b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:13:56.284 7fefd14f2b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.289 7fefda8f2708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:13:56.290 7fefd1dc2c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.290 7fefd1dc2c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:13:56.290 7fefd1dc2c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.302 7fefda8f2708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:13:56.342 7fefda8f2708 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:13:56.343 7fefd1534588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.343 7fefd1534588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:13:56.343 7fefd1534588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.355 7fefd1534b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.356 7fefd1534b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:13:56.356 7fefd1534b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.364 7fefd1541b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.364 7fefd1541b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:13:56.365 7fefd1541b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:56.365 7fefda8f2708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:13:56.365 7fefda8f2708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:13:56.365 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:13:56.367 7fefda8f2708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:13:56.384 7fefd1df0a20 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:13:56.397 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:13:56.404 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:13:56.422 7fefd1df0a20 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:13:56.429 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:13:56.881 7fefda8f2708 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:13:57.064 7fefda8f2708 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:13:57.246 7fefda8f2708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:13:57.247 7fefda8f2708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:13:57.431 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:57.442 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:13:57.464 7fefd1df0a20 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:13:57.476 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:13:57.748 7fefda8f2708 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:13:57.749 7fefda8f2708 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:13:57.749 7fefda8f2708 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:13:57.753 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:13:57.753 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:13:57.754 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:13:57.754 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:13:57.755 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:13:57.755 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:13:57.760 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:13:57.761 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:13:57.761 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:13:57.761 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:13:57.761 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:13:57.761 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:13:57.761 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:13:57.761 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:13:57.761 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:13:57.761 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:13:57.761 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:13:57.761 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:13:57.762 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:13:57.763 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:13:57.764 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:13:57.764 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:13:57.764 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:13:57.765 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:13:57.765 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:13:57.765 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:13:57.766 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:13:57.766 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:13:57.766 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:13:57.766 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:13:57.766 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:13:57.766 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:13:57.767 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:13:57.767 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:13:57.767 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:13:57.767 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:13:57.767 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:13:57.767 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:13:57.768 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:13:57.768 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:13:57.768 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:13:57.769 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:13:57.769 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:13:57.769 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:13:57.770 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:13:57.770 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:13:57.770 7fefda8f2708 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:13:57.775 7fefda8f2708 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:13:57.825 7fefd12f9f60 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:13:57.825 7fefd12f9f60 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:13:57.825 7fefd0be4438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.826 7fefd0be4438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:13:57.826 7fefd0be4438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.847 7fefd0bb14e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:57.847 7fefd0bb14e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:13:57.847 7fefd0bb14e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:13:58.479 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:13:58.491 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:13:58.513 7fefd1df0a20 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:13:58.543 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:13:59.549 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:00.097 7fefd0a90c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:14:00.110 7fefd0a90c18 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:14:00.112 7fefd0a90c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:14:00.112 7fefd0a90c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:14:00.112 7fefd0a90c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:14:00.112 7fefd0a90c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:14:00.112 7fefd0a90c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:14:00.112 7fefd0a90c18 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:14:00.113 7fefd0a90c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:14:00.113 7fefd0a90c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:14:00.113 7fefd0a90c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:14:00.113 7fefd0ac8978 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:14:00.114 7fefd0ac8978 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:14:00.128 7fefd0a90a20 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:14:00.128 7fefd0a90a20 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:14:00.129 7fefd2b81d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:14:00.129 7fefd2b81d68 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:14:00.129 7fefd2b81d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:14:00.129 7fefd2b81d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:14:00.130 7fefd2b81d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:14:00.130 7fefd0ac8978 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:14:00.132 7fefd0ac8978 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:14:00.550 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:01.551 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:02.553 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:03.553 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:04.555 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:05.556 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:06.557 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:07.558 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:08.559 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:09.560 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:10.561 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:11.562 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:12.565 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:13.566 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:14.568 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:15.569 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:16.569 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:17.570 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:18.571 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:19.572 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:20.574 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:21.575 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:22.577 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:23.578 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:24.578 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:25.588 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:26.598 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:27.605 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:28.611 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:29.617 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:30.619 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:31.621 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:32.625 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:33.626 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:34.629 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:35.630 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:36.631 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:37.632 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:38.633 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:39.634 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:40.635 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:41.636 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:42.637 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:43.638 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:44.639 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:45.640 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:46.642 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:47.643 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:48.645 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:48.978 7fefd0a97198 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:14:49.646 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:50.648 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:51.649 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:52.650 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:53.651 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:54.054 7fefd1dc2ac8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:14:54.652 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:55.653 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:56.654 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:57.656 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:58.657 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:14:59.659 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:00.660 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:01.662 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:02.663 7fefd1df0a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:02.835 7fefda8f2708 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:15:02.995 7fefda8f2708 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:15:42.053 7f923761d888 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:15:42.053 7f923761d888 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:15:42.054 7f923761d888 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:15:42.054 7f922ec886d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:15:42.054 7f922ec886d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:15:42.055 7f922ec886d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:15:42.055 7f922ec886d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:15:42.055 7f922ec886d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.055 7f922ec886d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:15:42.055 7f922ec886d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.055 7f922ec886d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.056 7f922ec886d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730945742055285 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:15:42.056 7f922ec886d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.091 7f923761d888 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:15:42.095 7f923761d888 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:15:42.096 7f922ebda6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.096 7f922ebda6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:15:42.096 7f922ebda6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.096 7f922ebda6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.097 7f922ebda6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:15:42.097 7f922ebda6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.109 7f923761d888 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:15:42.110 7f922ebd4048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.110 7f922ebd4048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:15:42.110 7f922ebd4048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.118 7f923761d888 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:15:42.119 7f922eb152e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.119 7f922eb152e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:15:42.119 7f922eb152e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.134 7f923761d888 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:15:42.134 7f922eb15978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.134 7f922eb15978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:15:42.135 7f922eb15978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.144 7f923761d888 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:15:42.144 7f923761d888 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:15:42.144 7f923761d888 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:15:42.145 7f923761d888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:15:42.145 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:15:42.145 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:15:42.146 7f922eb14a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:15:42.146 7f922eb14a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:15:42.146 7f922eb15d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.147 7f922eb15d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:15:42.147 7f922eb15d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.148 7f923761d888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:15:42.168 7f922eb14a20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:15:42.170 7f9234577780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:15:42.170 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:15:42.212 7f923761d888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:15:42.214 7f922e214b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.214 7f922e214b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:15:42.214 7f922e214b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.219 7f923761d888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:15:42.219 7f922eae4c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.219 7f922eae4c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:15:42.219 7f922eae4c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.231 7f923761d888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:15:42.275 7f923761d888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:15:42.276 7f922e200390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.276 7f922e200390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:15:42.276 7f922e200390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.289 7f922e200b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.290 7f922e200b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:15:42.290 7f922e200b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.298 7f922e25fb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.299 7f922e25fb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:15:42.299 7f922e25fb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:42.299 7f923761d888 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:15:42.299 7f923761d888 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:15:42.300 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:15:42.302 7f923761d888 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:15:42.314 7f9234577780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:15:42.322 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:15:42.330 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:15:42.346 7f9234577780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:15:42.354 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:15:42.821 7f923761d888 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:15:42.991 7f923761d888 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:15:43.186 7f923761d888 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:15:43.186 7f923761d888 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:15:43.357 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:43.365 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:15:43.382 7f9234577780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:15:43.400 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:15:43.687 7f923761d888 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:15:43.687 7f923761d888 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:15:43.687 7f923761d888 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:15:43.692 7f923761d888 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:15:43.692 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:15:43.693 7f923761d888 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:15:43.693 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:15:43.695 7f923761d888 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:15:43.695 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:15:43.700 7f923761d888 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:15:43.700 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:15:43.700 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:15:43.700 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:15:43.700 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:15:43.700 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:15:43.700 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:15:43.700 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:15:43.700 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:15:43.701 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:15:43.701 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:15:43.701 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:15:43.701 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:15:43.702 7f923761d888 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:15:43.703 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:15:43.703 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:15:43.703 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:15:43.704 7f923761d888 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:15:43.704 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:15:43.704 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:15:43.704 7f923761d888 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:15:43.706 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:15:43.706 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:15:43.706 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:15:43.706 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:15:43.707 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:15:43.707 7f923761d888 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:15:43.708 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:15:43.708 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:15:43.708 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:15:43.708 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:15:43.708 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:15:43.708 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:15:43.708 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:15:43.708 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:15:43.709 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:15:43.709 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:15:43.709 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:15:43.709 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:15:43.709 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:15:43.709 7f923761d888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:15:43.714 7f923761d888 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:15:43.765 7f922e0c0cc0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:15:43.765 7f922e0c0cc0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:15:43.766 7f922d8fd4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:43.766 7f922d8fd4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:15:43.766 7f922d8fd4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:43.792 7f922d8d5588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:43.793 7f922d8d5588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:15:43.793 7f922d8d5588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:15:44.402 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:44.411 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:15:44.428 7f9234577780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:15:44.467 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:15:45.472 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:46.054 7f922d7baac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:15:46.062 7f922d7baac8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:15:46.063 7f922d7baac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:15:46.063 7f922d7baac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:15:46.063 7f922d7baac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:15:46.063 7f922d7baac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:15:46.063 7f922d7baac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:15:46.064 7f922d7baac8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:15:46.064 7f922d7baac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:15:46.064 7f922d7baac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:15:46.064 7f922d7baac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:15:46.065 7f922d7c80f0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:15:46.066 7f922d7c80f0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:15:46.069 7f922d7b3eb8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:15:46.069 7f922d7b3eb8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:15:46.069 7f922f83be10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:15:46.069 7f922f83be10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:15:46.070 7f922f83be10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:15:46.070 7f922f83be10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:15:46.070 7f922f83be10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:15:46.070 7f922d7baac8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:15:46.072 7f922d7baac8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:15:46.473 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:47.475 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:48.476 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:49.477 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:50.478 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:51.479 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:52.480 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:53.481 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:54.482 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:55.483 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:56.484 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:57.485 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:58.487 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:15:59.488 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:00.488 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:01.490 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:02.490 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:03.491 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:04.492 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:05.494 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:06.495 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:07.496 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:08.500 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:09.501 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:10.502 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:11.503 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:12.504 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:13.506 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:14.507 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:15.509 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:16.510 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:17.511 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:18.512 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:19.513 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:20.514 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:21.515 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:22.517 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:23.518 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:24.520 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:25.521 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:26.522 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:27.524 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:28.525 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:29.527 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:30.528 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:31.530 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:32.531 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:33.532 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:34.533 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:35.534 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:36.535 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:37.537 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:38.537 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:39.539 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:40.540 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:41.541 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:42.542 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:43.544 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:44.545 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:16:45.371 7f923761d888 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:16:45.535 7f923761d888 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:16:45.546 7f9234577780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:28.034 7fc6faa39288 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:17:28.034 7fc6faa39288 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:17:28.034 7fc6faa39288 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:17:28.035 7fc6f20d26d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:17:28.035 7fc6f20d26d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:17:28.036 7fc6f20d26d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:17:28.036 7fc6f20d26d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:17:28.036 7fc6f20d26d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.036 7fc6f20d26d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:17:28.036 7fc6f20d26d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.036 7fc6f20d26d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.037 7fc6f20d26d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730945848036317 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:17:28.037 7fc6f20d26d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.076 7fc6faa39288 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:17:28.081 7fc6faa39288 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:17:28.082 7fc6f20226d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.083 7fc6f20226d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:17:28.083 7fc6f20226d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.083 7fc6f20226d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.083 7fc6f20226d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:17:28.083 7fc6f20226d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.096 7fc6faa39288 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:17:28.096 7fc6f201c048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.097 7fc6f201c048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:17:28.097 7fc6f201c048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.105 7fc6faa39288 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:17:28.106 7fc6f1f5e2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.106 7fc6f1f5e2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:17:28.106 7fc6f1f5e2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.119 7fc6faa39288 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:17:28.120 7fc6f1f5e978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.120 7fc6f1f5e978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:17:28.120 7fc6f1f5e978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.128 7fc6faa39288 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:17:28.128 7fc6faa39288 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:17:28.128 7fc6faa39288 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:17:28.129 7fc6faa39288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:17:28.129 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:17:28.129 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:17:28.130 7fc6f1f5ca20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:17:28.130 7fc6f1f5ca20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:17:28.130 7fc6f1f5ed68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.130 7fc6f1f5ed68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:17:28.130 7fc6f1f5ed68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.131 7fc6faa39288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:17:28.147 7fc6f1f5ca20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:17:28.147 7fc6f79be780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:17:28.148 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:17:28.186 7fc6faa39288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:17:28.188 7fc6f1e59d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.188 7fc6f1e59d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:17:28.188 7fc6f1e59d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.192 7fc6faa39288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:17:28.193 7fc6f1eae978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.193 7fc6f1eae978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:17:28.193 7fc6f1eae978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.206 7fc6faa39288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:17:28.253 7fc6faa39288 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:17:28.253 7fc6f1e5f0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.254 7fc6f1e5f0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:17:28.254 7fc6f1e5f0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.268 7fc6f1e5f8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.268 7fc6f1e5f8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:17:28.268 7fc6f1e5f8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.277 7fc6f1ea68d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.277 7fc6f1ea68d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:17:28.278 7fc6f1ea68d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:28.278 7fc6faa39288 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:17:28.278 7fc6faa39288 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:17:28.278 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:17:28.278 7fc6faa39288 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:17:28.291 7fc6f79be780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:17:28.298 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:17:28.306 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:17:28.320 7fc6f79be780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:17:28.328 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:17:28.791 7fc6faa39288 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:17:28.980 7fc6faa39288 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:17:29.173 7fc6faa39288 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:17:29.174 7fc6faa39288 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:17:29.331 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:29.338 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:17:29.351 7fc6f79be780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:17:29.365 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:17:29.675 7fc6faa39288 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:17:29.675 7fc6faa39288 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:17:29.675 7fc6faa39288 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:17:29.679 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:17:29.679 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:17:29.680 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:17:29.680 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:17:29.681 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:17:29.681 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:17:29.685 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:17:29.685 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:17:29.685 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:17:29.685 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:17:29.686 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:17:29.686 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:17:29.686 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:17:29.686 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:17:29.686 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:17:29.686 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:17:29.686 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:17:29.686 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:17:29.686 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:17:29.688 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:17:29.688 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:17:29.688 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:17:29.688 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:17:29.688 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:17:29.688 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:17:29.689 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:17:29.689 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:17:29.689 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:17:29.689 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:17:29.689 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:17:29.689 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:17:29.689 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:17:29.690 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:17:29.690 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:17:29.690 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:17:29.691 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:17:29.691 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:17:29.691 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:17:29.691 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:17:29.691 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:17:29.691 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:17:29.691 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:17:29.691 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:17:29.691 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:17:29.691 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:17:29.692 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:17:29.692 7fc6faa39288 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:17:29.694 7fc6faa39288 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:17:29.739 7fc6f1d19f60 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:17:29.739 7fc6f1d19f60 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:17:29.739 7fc6f14d7198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:29.740 7fc6f14d7198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:17:29.740 7fc6f14d7198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:29.767 7fc6f152d240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:29.768 7fc6f152d240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:17:29.768 7fc6f152d240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:17:30.374 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:30.383 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:17:30.401 7fc6f79be780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:17:30.420 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:17:31.428 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:32.036 7fc6f0bfcc18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:17:32.044 7fc6f0bfcc18 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:17:32.044 7fc6f0bfcc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:17:32.044 7fc6f0bfcc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:17:32.045 7fc6f0bfcc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:17:32.045 7fc6f0bfcc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:17:32.045 7fc6f0bfcc18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:17:32.045 7fc6f0bfcc18 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:17:32.046 7fc6f0bfcc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:17:32.046 7fc6f0bfcc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:17:32.046 7fc6f0bfcc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:17:32.046 7fc6f0c12438 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:17:32.047 7fc6f0c12438 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:17:32.050 7fc6f0c122e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:17:32.050 7fc6f0c122e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:17:32.051 7fc6f2c84e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:17:32.051 7fc6f2c84e10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:17:32.051 7fc6f2c84e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:17:32.051 7fc6f2c84e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:17:32.051 7fc6f2c84e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:17:32.051 7fc6f0c12048 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:17:32.053 7fc6f0c12048 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:17:32.430 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:33.431 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:34.431 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:35.433 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:36.434 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:37.436 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:38.437 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:39.438 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:40.439 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:41.440 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:42.441 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:43.442 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:44.443 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:45.444 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:46.445 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:47.446 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:48.447 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:49.448 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:50.449 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:51.450 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:52.451 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:53.453 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:54.454 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:55.455 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:56.457 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:57.458 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:58.459 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:17:59.459 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:00.461 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:01.462 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:02.463 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:03.464 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:04.466 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:05.467 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:06.469 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:07.470 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:08.471 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:09.472 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:10.473 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:11.474 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:12.475 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:13.476 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:14.477 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:15.479 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:16.480 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:17.481 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:18.482 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:19.483 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:20.484 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:21.485 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:22.486 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:23.487 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:24.489 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:25.490 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:26.491 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:27.492 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:28.493 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:29.494 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:30.495 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:18:30.999 7fc6faa39288 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:18:31.167 7fc6faa39288 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:18:31.635 7fc6f79be780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:14.093 7fa5fb691548 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:19:14.093 7fa5fb691548 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:19:14.094 7fa5fb691548 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:19:14.094 7fa5f2cc26d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:19:14.094 7fa5f2cc26d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:19:14.094 7fa5f2cc26d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:19:14.095 7fa5f2cc26d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:19:14.095 7fa5f2cc26d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.095 7fa5f2cc26d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:19:14.095 7fa5f2cc26d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.095 7fa5f2cc26d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.095 7fa5f2cc26d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730945954094905 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:19:14.095 7fa5f2cc26d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.132 7fa5fb691548 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:19:14.137 7fa5fb691548 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:19:14.137 7fa5f2c116d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.138 7fa5f2c116d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:19:14.138 7fa5f2c116d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.138 7fa5f2c116d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.138 7fa5f2c116d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:19:14.138 7fa5f2c116d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.152 7fa5fb691548 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:19:14.153 7fa5f2c0a048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.153 7fa5f2c0a048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:19:14.153 7fa5f2c0a048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.161 7fa5fb691548 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:19:14.162 7fa5f2b4c2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.162 7fa5f2b4c2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:19:14.162 7fa5f2b4c2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.174 7fa5fb691548 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:19:14.175 7fa5f2b4c978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.175 7fa5f2b4c978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:19:14.175 7fa5f2b4c978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.184 7fa5fb691548 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:19:14.185 7fa5fb691548 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:19:14.185 7fa5fb691548 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:19:14.185 7fa5fb691548 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:19:14.186 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:19:14.186 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:19:14.186 7fa5f2b4ba20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:19:14.187 7fa5f2b4ba20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:19:14.187 7fa5f2b4cd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.188 7fa5f2b4cd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:19:14.188 7fa5f2b4cd68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.189 7fa5fb691548 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:19:14.206 7fa5f85ae780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:19:14.206 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:19:14.211 7fa5f2b4ba20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:19:14.249 7fa5fb691548 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:19:14.251 7fa5f224ab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.251 7fa5f224ab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:19:14.251 7fa5f224ab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.256 7fa5fb691548 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:19:14.256 7fa5f22e4c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.256 7fa5f22e4c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:19:14.256 7fa5f22e4c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.269 7fa5fb691548 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:19:14.312 7fa5fb691548 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:19:14.313 7fa5f228c390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.313 7fa5f228c390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:19:14.313 7fa5f228c390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.326 7fa5f228cb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.327 7fa5f228cb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:19:14.327 7fa5f228cb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.335 7fa5f2297b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.336 7fa5f2297b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:19:14.336 7fa5f2297b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:14.336 7fa5fb691548 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:19:14.336 7fa5fb691548 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:19:14.336 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:19:14.337 7fa5fb691548 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:19:14.351 7fa5f85ae780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:19:14.360 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:19:14.367 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:19:14.394 7fa5f85ae780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:19:14.402 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:19:14.853 7fa5fb691548 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:19:15.026 7fa5fb691548 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:19:15.226 7fa5fb691548 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:19:15.227 7fa5fb691548 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:19:15.404 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:15.413 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:19:15.432 7fa5f85ae780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:19:15.444 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:19:15.728 7fa5fb691548 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:19:15.728 7fa5fb691548 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:19:15.728 7fa5fb691548 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:19:15.734 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:19:15.735 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:19:15.736 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:19:15.736 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:19:15.737 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:19:15.737 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:19:15.742 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:19:15.742 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:19:15.742 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:19:15.742 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:19:15.742 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:19:15.742 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:19:15.742 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:19:15.742 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:19:15.742 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:19:15.742 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:19:15.742 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:19:15.742 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:19:15.743 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:19:15.744 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:19:15.744 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:19:15.745 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:19:15.745 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:19:15.745 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:19:15.745 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:19:15.745 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:19:15.746 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:19:15.746 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:19:15.746 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:19:15.746 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:19:15.746 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:19:15.746 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:19:15.747 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:19:15.747 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:19:15.747 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:19:15.748 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:19:15.748 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:19:15.748 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:19:15.748 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:19:15.748 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:19:15.748 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:19:15.748 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:19:15.748 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:19:15.748 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:19:15.749 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:19:15.749 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:19:15.749 7fa5fb691548 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:19:15.751 7fa5fb691548 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:19:15.798 7fa5f2107198 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:19:15.798 7fa5f2107198 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:19:15.798 7fa5f193f438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:15.799 7fa5f193f438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:19:15.799 7fa5f193f438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:15.821 7fa5f190b4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:15.821 7fa5f190b4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:19:15.821 7fa5f190b4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:19:16.447 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:16.454 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:19:16.470 7fa5f85ae780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:19:16.484 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:19:17.488 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:18.094 7fa5f17eac18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:19:18.102 7fa5f17eac18 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:19:18.102 7fa5f17eac18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:19:18.102 7fa5f17eac18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:19:18.102 7fa5f17eac18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:19:18.102 7fa5f17eac18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:19:18.102 7fa5f17eac18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:19:18.102 7fa5f17eac18 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:19:18.103 7fa5f17eac18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:19:18.103 7fa5f17eac18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:19:18.103 7fa5f17eac18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:19:18.103 7fa5f17fe438 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:19:18.104 7fa5f17fe438 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:19:18.107 7fa5f17fe2e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:19:18.108 7fa5f17fe2e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:19:18.108 7fa5f38dbd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:19:18.108 7fa5f38dbd68 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:19:18.109 7fa5f38dbd68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:19:18.109 7fa5f38dbd68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:19:18.109 7fa5f38dbd68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:19:18.109 7fa5f17fe048 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:19:18.110 7fa5f17fe048 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:19:18.489 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:19.489 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:20.490 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:21.491 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:22.493 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:23.494 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:24.495 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:25.496 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:26.498 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:27.499 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:28.500 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:29.501 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:30.502 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:31.504 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:32.505 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:33.506 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:34.507 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:35.509 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:36.510 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:37.511 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:38.512 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:39.513 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:40.514 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:41.517 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:42.518 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:43.519 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:44.521 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:45.521 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:46.522 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:47.524 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:48.525 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:49.526 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:50.527 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:51.528 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:52.021 7fa5f228c0f0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:19:52.530 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:53.531 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:54.532 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:55.534 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:56.535 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:57.537 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:57.606 7fa5f877de10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:19:58.538 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:19:59.539 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:00.540 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:01.540 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:02.541 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:03.542 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:04.543 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:05.545 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:06.546 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:07.547 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:08.548 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:09.549 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:10.551 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:11.552 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:12.553 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:13.554 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:14.556 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:15.559 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:16.561 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:17.563 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:18.223 7fa5fb691548 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:20:18.419 7fa5fb691548 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:20:18.572 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:20:19.636 7fa5f85ae780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:00.051 7f549e5fd748 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:21:00.051 7f549e5fd748 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:21:00.052 7f549e5fd748 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:21:00.052 7f5495c716d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:21:00.052 7f5495c716d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:21:00.053 7f5495c716d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:21:00.053 7f5495c716d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:21:00.053 7f5495c716d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.053 7f5495c716d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:21:00.053 7f5495c716d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.053 7f5495c716d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.053 7f5495c716d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730946060053042 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:21:00.053 7f5495c716d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.100 7f549e5fd748 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:21:00.105 7f549e5fd748 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:21:00.106 7f5495bc36d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.107 7f5495bc36d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:21:00.107 7f5495bc36d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.107 7f5495bc36d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.107 7f5495bc36d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:21:00.108 7f5495bc36d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.136 7f549e5fd748 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:21:00.137 7f5495bbb048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.137 7f5495bbb048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:21:00.137 7f5495bbb048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.147 7f549e5fd748 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:21:00.148 7f5495afe2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.148 7f5495afe2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:21:00.149 7f5495afe2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.164 7f549e5fd748 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:21:00.164 7f5495afe978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.165 7f5495afe978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:21:00.165 7f5495afe978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.174 7f549e5fd748 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:21:00.175 7f549e5fd748 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:21:00.175 7f549e5fd748 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:21:00.176 7f549e5fd748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:21:00.176 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:21:00.176 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:21:00.177 7f5495af9a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:21:00.177 7f5495af9a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:21:00.177 7f5495afed68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.178 7f5495afed68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:21:00.178 7f5495afed68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.179 7f549e5fd748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:21:00.198 7f5495af9a20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:21:00.201 7f549b561780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:21:00.201 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:21:00.255 7f549e5fd748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:21:00.257 7f54951fbb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.257 7f54951fbb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:21:00.257 7f54951fbb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.263 7f549e5fd748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:21:00.264 7f5495298c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.264 7f5495298c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:21:00.264 7f5495298c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.278 7f549e5fd748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:21:00.323 7f549e5fd748 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:21:00.324 7f549523c390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.324 7f549523c390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:21:00.324 7f549523c390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.339 7f549523cb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.340 7f549523cb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:21:00.340 7f549523cb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.354 7f5495249b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.355 7f5495249b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:21:00.355 7f5495249b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:00.356 7f549e5fd748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:21:00.356 7f549e5fd748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:21:00.356 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:21:00.356 7f549e5fd748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:21:00.378 7f549b561780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:21:00.388 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:21:00.399 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:21:00.417 7f549b561780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:21:00.434 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:21:00.877 7f549e5fd748 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:21:01.055 7f549e5fd748 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:21:01.252 7f549e5fd748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:21:01.252 7f549e5fd748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:21:01.437 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:01.445 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:21:01.462 7f549b561780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:21:01.475 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:21:01.753 7f549e5fd748 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:21:01.753 7f549e5fd748 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:21:01.753 7f549e5fd748 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:21:01.757 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:21:01.757 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:21:01.759 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:21:01.759 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:21:01.760 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:21:01.760 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:21:01.764 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:21:01.765 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:21:01.765 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:21:01.765 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:21:01.765 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:21:01.765 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:21:01.765 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:21:01.765 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:21:01.765 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:21:01.765 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:21:01.765 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:21:01.766 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:21:01.766 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:21:01.767 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:21:01.767 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:21:01.768 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:21:01.768 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:21:01.768 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:21:01.768 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:21:01.768 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:21:01.769 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:21:01.769 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:21:01.769 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:21:01.769 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:21:01.769 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:21:01.769 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:21:01.770 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:21:01.770 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:21:01.770 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:21:01.771 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:21:01.771 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:21:01.771 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:21:01.771 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:21:01.771 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:21:01.771 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:21:01.771 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:21:01.771 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:21:01.771 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:21:01.772 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:21:01.772 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:21:01.772 7f549e5fd748 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:21:01.775 7f549e5fd748 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:21:01.823 7f549510d390 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:21:01.823 7f549510d390 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:21:01.824 7f54948ef438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:01.824 7f54948ef438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:21:01.824 7f54948ef438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:01.847 7f54948ba4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:01.847 7f54948ba4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:21:01.847 7f54948ba4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:21:02.479 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:02.487 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:21:02.501 7f549b561780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:21:02.514 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:21:03.519 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:04.053 7f5494799c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:21:04.060 7f5494799c18 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:21:04.061 7f5494799c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:21:04.061 7f5494799c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:21:04.061 7f5494799c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:21:04.061 7f5494799c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:21:04.061 7f5494799c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:21:04.061 7f5494799c18 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:21:04.062 7f5494799c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:21:04.062 7f5494799c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:21:04.062 7f5494799c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:21:04.062 7f54947b3438 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:21:04.064 7f54947b3438 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:21:04.067 7f54947b32e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:21:04.067 7f54947b32e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:21:04.068 7f549688ed68 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:21:04.068 7f549688ed68 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:21:04.068 7f549688ed68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:21:04.068 7f549688ed68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:21:04.068 7f549688ed68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:21:04.069 7f54947b3048 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:21:04.070 7f54947b3048 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:21:04.520 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:05.521 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:06.522 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:07.527 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:08.528 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:09.530 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:10.531 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:11.532 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:11.903 7f54947a1390 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:21:12.533 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:13.534 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:14.537 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:15.538 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:16.539 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:17.540 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:18.542 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:19.543 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:20.544 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:21.546 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:22.547 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:23.549 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:24.549 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:25.550 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:26.551 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:27.553 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:28.554 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:29.555 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:30.556 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:31.558 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:32.559 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:33.561 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:34.562 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:35.563 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:36.564 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:37.565 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:38.566 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:39.567 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:40.568 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:41.569 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:42.569 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:43.571 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:44.572 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:45.573 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:46.574 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:47.575 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:48.576 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:49.578 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:50.579 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:51.581 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:52.581 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:53.583 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:54.583 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:55.585 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:56.586 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:57.593 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:58.594 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:21:59.595 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:00.596 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:01.597 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:02.597 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:03.367 7f549e5fd748 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:22:03.523 7f549e5fd748 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:22:03.599 7f549b561780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:46.085 7f06177112c8 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:22:46.085 7f06177112c8 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:22:46.085 7f06177112c8 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:22:46.085 7f060ed936d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:22:46.085 7f060ed936d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:22:46.086 7f060ed936d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:22:46.086 7f060ed936d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:22:46.086 7f060ed936d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.087 7f060ed936d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:22:46.087 7f060ed936d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.087 7f060ed936d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.087 7f060ed936d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730946166086361 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:22:46.087 7f060ed936d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.127 7f06177112c8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:22:46.133 7f06177112c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:22:46.134 7f060ece66d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.134 7f060ece66d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:22:46.134 7f060ece66d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.134 7f060ece66d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.134 7f060ece66d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:22:46.134 7f060ece66d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.148 7f06177112c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:22:46.149 7f060ecdf048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.149 7f060ecdf048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:22:46.149 7f060ecdf048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.157 7f06177112c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:22:46.158 7f060ec222e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.159 7f060ec222e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:22:46.159 7f060ec222e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.171 7f06177112c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:22:46.172 7f060ec22978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.172 7f060ec22978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:22:46.172 7f060ec22978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.181 7f06177112c8 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:22:46.181 7f06177112c8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:22:46.182 7f06177112c8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:22:46.182 7f06177112c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:22:46.182 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:22:46.183 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:22:46.183 7f060ec20a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:22:46.183 7f060ec20a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:22:46.184 7f060ec22d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.184 7f060ec22d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:22:46.184 7f060ec22d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.186 7f06177112c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:22:46.203 7f060ec20a20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:22:46.203 7f0614685780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:22:46.204 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:22:46.243 7f06177112c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:22:46.246 7f060eb1dd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.247 7f060eb1dd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:22:46.247 7f060eb1dd68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.255 7f06177112c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:22:46.255 7f060eb72978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.255 7f060eb72978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:22:46.256 7f060eb72978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.268 7f06177112c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:22:46.311 7f06177112c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:22:46.312 7f060eb240f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.312 7f060eb240f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:22:46.313 7f060eb240f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.326 7f060eb248d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.327 7f060eb248d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:22:46.327 7f060eb248d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.335 7f060eb6c8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.335 7f060eb6c8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:22:46.336 7f060eb6c8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:46.336 7f06177112c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:22:46.336 7f06177112c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:22:46.336 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:22:46.337 7f06177112c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:22:46.351 7f0614685780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:22:46.361 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:22:46.368 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:22:46.382 7f0614685780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:22:46.390 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:22:46.852 7f06177112c8 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:22:47.014 7f06177112c8 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:22:47.187 7f06177112c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:22:47.188 7f06177112c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:22:47.392 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:47.404 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:22:47.419 7f0614685780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:22:47.429 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:22:47.688 7f06177112c8 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:22:47.688 7f06177112c8 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:22:47.689 7f06177112c8 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:22:47.692 7f06177112c8 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:22:47.692 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:22:47.693 7f06177112c8 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:22:47.693 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:22:47.695 7f06177112c8 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:22:47.695 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:22:47.699 7f06177112c8 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:22:47.699 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:22:47.700 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:22:47.700 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:22:47.700 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:22:47.700 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:22:47.700 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:22:47.700 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:22:47.700 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:22:47.700 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:22:47.700 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:22:47.700 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:22:47.700 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:22:47.702 7f06177112c8 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:22:47.702 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:22:47.703 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:22:47.703 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:22:47.703 7f06177112c8 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:22:47.703 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:22:47.703 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:22:47.704 7f06177112c8 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:22:47.704 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:22:47.704 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:22:47.704 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:22:47.704 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:22:47.704 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:22:47.705 7f06177112c8 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:22:47.705 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:22:47.705 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:22:47.706 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:22:47.706 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:22:47.706 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:22:47.706 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:22:47.706 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:22:47.706 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:22:47.706 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:22:47.707 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:22:47.707 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:22:47.707 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:22:47.707 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:22:47.707 7f06177112c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:22:47.710 7f06177112c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:22:47.757 7f060e9ddf60 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:22:47.758 7f060e9ddf60 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:22:47.758 7f060e19b198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:47.758 7f060e19b198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:22:47.758 7f060e19b198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:47.849 7f060e1f0240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:47.849 7f060e1f0240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:22:47.849 7f060e1f0240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:22:48.432 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:48.440 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:22:48.455 7f0614685780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:22:48.470 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:22:49.473 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:50.087 7f060d8bdc18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:22:50.098 7f060d8bdc18 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:22:50.099 7f060d8bdc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:22:50.099 7f060d8bdc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:22:50.099 7f060d8bdc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:22:50.099 7f060d8bdc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:22:50.099 7f060d8bdc18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:22:50.100 7f060d8bdc18 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:22:50.100 7f060d8bdc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:22:50.100 7f060d8bdc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:22:50.101 7f060d8bdc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:22:50.101 7f060d8d5438 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:22:50.103 7f060d8d5438 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:22:50.105 7f060d8d52e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:22:50.106 7f060d8d52e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:22:50.106 7f060f947e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:22:50.106 7f060f947e10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:22:50.106 7f060f947e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:22:50.107 7f060f947e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:22:50.107 7f060f947e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:22:50.107 7f060d8d5048 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:22:50.109 7f060d8d5048 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:22:50.474 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:51.476 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:52.476 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:53.477 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:54.478 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:55.479 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:56.480 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:57.480 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:58.482 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:22:59.483 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:00.484 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:01.485 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:02.486 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:03.487 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:04.488 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:05.489 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:06.490 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:07.492 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:08.493 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:09.494 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:10.495 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:11.496 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:12.497 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:13.499 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:14.500 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:15.500 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:16.501 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:17.503 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:18.503 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:19.505 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:20.506 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:21.507 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:22.508 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:23.509 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:24.511 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:25.512 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:26.513 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:27.514 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:28.515 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:29.516 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:30.518 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:31.519 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:32.520 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:33.522 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:34.523 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:35.524 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:36.525 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:37.526 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:38.527 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:39.528 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:40.530 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:41.531 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:42.532 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:43.533 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:44.533 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:45.535 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:46.536 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:47.537 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:48.538 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:23:49.107 7f06177112c8 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:23:49.267 7f06177112c8 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:23:49.540 7f0614685780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:32.030 7ff205cd8508 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:24:32.030 7ff205cd8508 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:24:32.030 7ff205cd8508 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:24:32.030 7ff1fd3a56d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:24:32.030 7ff1fd3a56d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:24:32.031 7ff1fd3a56d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:24:32.031 7ff1fd3a56d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:24:32.031 7ff1fd3a56d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.031 7ff1fd3a56d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:24:32.032 7ff1fd3a56d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.032 7ff1fd3a56d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.032 7ff1fd3a56d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730946272031291 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:24:32.032 7ff1fd3a56d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.074 7ff205cd8508 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:24:32.080 7ff205cd8508 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:24:32.081 7ff1fd2f46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.081 7ff1fd2f46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:24:32.082 7ff1fd2f46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.082 7ff1fd2f46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.082 7ff1fd2f46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:24:32.082 7ff1fd2f46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.095 7ff205cd8508 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:24:32.096 7ff1fd2ee048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.096 7ff1fd2ee048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:24:32.096 7ff1fd2ee048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.103 7ff205cd8508 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:24:32.104 7ff1fd22f2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.104 7ff1fd22f2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:24:32.105 7ff1fd22f2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.119 7ff205cd8508 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:24:32.120 7ff1fd22f978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.120 7ff1fd22f978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:24:32.120 7ff1fd22f978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.128 7ff205cd8508 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:24:32.129 7ff205cd8508 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:24:32.129 7ff205cd8508 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:24:32.129 7ff205cd8508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:24:32.130 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:24:32.130 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:24:32.131 7ff1fd22da20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:24:32.131 7ff1fd22da20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:24:32.131 7ff1fd22fd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.132 7ff1fd22fd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:24:32.132 7ff1fd22fd68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.133 7ff205cd8508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:24:32.150 7ff1fd22da20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:24:32.155 7ff202c91780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:24:32.155 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:24:32.194 7ff205cd8508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:24:32.196 7ff1fd12ad68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.196 7ff1fd12ad68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:24:32.196 7ff1fd12ad68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.201 7ff205cd8508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:24:32.201 7ff1fd181978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.201 7ff1fd181978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:24:32.201 7ff1fd181978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.213 7ff205cd8508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:24:32.257 7ff205cd8508 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:24:32.258 7ff1fd16f0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.258 7ff1fd16f0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:24:32.258 7ff1fd16f0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.271 7ff1fd16f8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.271 7ff1fd16f8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:24:32.271 7ff1fd16f8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.279 7ff1fd17a8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.279 7ff1fd17a8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:24:32.279 7ff1fd17a8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:32.280 7ff205cd8508 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:24:32.280 7ff205cd8508 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:24:32.280 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:24:32.281 7ff205cd8508 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:24:32.294 7ff202c91780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:24:32.303 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:24:32.311 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:24:32.325 7ff202c91780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:24:32.333 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:24:32.794 7ff205cd8508 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:24:32.957 7ff205cd8508 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:24:33.126 7ff205cd8508 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:24:33.126 7ff205cd8508 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:24:33.336 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:33.344 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:24:33.359 7ff202c91780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:24:33.370 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:24:33.627 7ff205cd8508 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:24:33.627 7ff205cd8508 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:24:33.627 7ff205cd8508 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:24:33.632 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:24:33.632 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:24:33.633 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:24:33.633 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:24:33.634 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:24:33.634 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:24:33.638 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:24:33.639 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:24:33.639 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:24:33.639 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:24:33.639 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:24:33.639 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:24:33.639 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:24:33.639 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:24:33.639 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:24:33.639 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:24:33.639 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:24:33.639 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:24:33.639 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:24:33.641 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:24:33.641 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:24:33.641 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:24:33.641 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:24:33.642 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:24:33.642 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:24:33.642 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:24:33.643 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:24:33.643 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:24:33.643 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:24:33.643 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:24:33.643 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:24:33.643 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:24:33.644 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:24:33.644 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:24:33.644 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:24:33.644 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:24:33.644 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:24:33.644 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:24:33.644 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:24:33.645 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:24:33.645 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:24:33.645 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:24:33.645 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:24:33.645 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:24:33.645 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:24:33.645 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:24:33.645 7ff205cd8508 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:24:33.648 7ff205cd8508 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:24:33.710 7ff1fcfeb390 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:24:33.710 7ff1fcfeb390 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:24:33.711 7ff1fc7a8198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:33.711 7ff1fc7a8198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:24:33.711 7ff1fc7a8198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:33.733 7ff1fc7ee240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:33.733 7ff1fc7ee240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:24:33.733 7ff1fc7ee240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:24:34.374 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:34.381 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:24:34.395 7ff202c91780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:24:34.410 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:24:35.413 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:36.031 7ff1fc6d38d0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:24:36.042 7ff1fc6d38d0 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:24:36.042 7ff1fc6d38d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:24:36.043 7ff1fc6d38d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:36.043 7ff1fc6d38d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:24:36.043 7ff1fc6d38d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:24:36.043 7ff1fc6d38d0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:24:36.043 7ff1fc6d38d0 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:24:36.044 7ff1fc6d38d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:24:36.044 7ff1fc6d38d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:24:36.044 7ff1fc6d38d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:24:36.045 7ff1fc667198 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:24:36.046 7ff1fc667198 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:24:36.053 7ff1fc667048 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:24:36.053 7ff1fc667048 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:24:36.054 7ff1fdfbfd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:24:36.054 7ff1fdfbfd68 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:24:36.054 7ff1fdfbfd68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:24:36.055 7ff1fdfbfd68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:24:36.055 7ff1fdfbfd68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:24:36.055 7ff1fc6d38d0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:24:36.056 7ff1fc6d38d0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:24:36.414 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:37.415 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:38.416 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:39.418 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:40.419 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:41.419 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:42.421 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:43.421 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:44.422 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:45.423 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:46.424 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:47.425 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:48.426 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:49.428 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:50.429 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:51.430 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:52.435 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:53.436 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:54.437 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:55.469 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:56.470 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:57.471 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:58.503 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:24:59.504 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:00.505 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:00.732 7ff1fc6d3978 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:25:01.506 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:02.507 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:03.509 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:04.510 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:05.511 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:06.512 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:07.513 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:08.514 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:09.515 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:10.516 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:11.517 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:12.517 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:13.519 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:14.520 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:15.521 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:16.522 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:17.523 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:18.526 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:19.527 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:20.367 7ff204b59e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:25:20.528 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:21.529 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:22.530 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:23.531 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:24.532 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:25.533 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:26.534 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:27.537 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:28.538 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:29.539 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:30.541 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:31.542 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:32.543 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:33.543 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:34.545 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:35.546 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:36.547 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:37.548 7ff202c91780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:25:37.787 7ff205cd8508 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:25:37.947 7ff205cd8508 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:26:18.055 7f1edf75f088 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:26:18.056 7f1edf75f088 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:26:18.056 7f1edf75f088 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:26:18.056 7f1ed6dc46d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:26:18.056 7f1ed6dc46d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:26:18.057 7f1ed6dc46d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:26:18.057 7f1ed6dc46d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:26:18.057 7f1ed6dc46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.058 7f1ed6dc46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:26:18.058 7f1ed6dc46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.058 7f1ed6dc46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.058 7f1ed6dc46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730946378057463 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:26:18.058 7f1ed6dc46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.102 7f1edf75f088 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:26:18.108 7f1edf75f088 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:26:18.108 7f1ed6d136d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.110 7f1ed6d136d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:26:18.110 7f1ed6d136d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.110 7f1ed6d136d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.110 7f1ed6d136d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:26:18.111 7f1ed6d136d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.123 7f1edf75f088 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:26:18.124 7f1ed6d0d048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.124 7f1ed6d0d048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:26:18.124 7f1ed6d0d048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.131 7f1edf75f088 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:26:18.132 7f1ed6c512e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.132 7f1ed6c512e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:26:18.132 7f1ed6c512e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.145 7f1edf75f088 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:26:18.146 7f1ed6c51978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.146 7f1ed6c51978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:26:18.146 7f1ed6c51978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.153 7f1edf75f088 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:26:18.154 7f1edf75f088 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:26:18.154 7f1edf75f088 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:26:18.155 7f1edf75f088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:26:18.155 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:26:18.155 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:26:18.156 7f1ed6c4ba20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:26:18.156 7f1ed6c4ba20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:26:18.157 7f1ed6c51d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.157 7f1ed6c51d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:26:18.157 7f1ed6c51d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.159 7f1edf75f088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:26:18.179 7f1edc6b3780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:26:18.180 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:26:18.180 7f1ed6c4ba20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:26:18.212 7f1edf75f088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:26:18.214 7f1ed634eb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.214 7f1ed634eb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:26:18.214 7f1ed634eb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.219 7f1edf75f088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:26:18.219 7f1ed63b5c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.219 7f1ed63b5c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:26:18.219 7f1ed63b5c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.231 7f1edf75f088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:26:18.268 7f1edf75f088 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:26:18.268 7f1ed638f390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.269 7f1ed638f390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:26:18.269 7f1ed638f390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.284 7f1ed638fb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.284 7f1ed638fb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:26:18.284 7f1ed638fb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.292 7f1ed6397b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.292 7f1ed6397b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:26:18.293 7f1ed6397b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:18.293 7f1edf75f088 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:26:18.293 7f1edf75f088 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:26:18.293 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:26:18.294 7f1edf75f088 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:26:18.307 7f1edc6b3780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:26:18.315 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:26:18.323 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:26:18.338 7f1edc6b3780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:26:18.346 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:26:18.806 7f1edf75f088 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:26:18.967 7f1edf75f088 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:26:19.157 7f1edf75f088 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:26:19.157 7f1edf75f088 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:26:19.349 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:19.357 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:26:19.372 7f1edc6b3780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:26:19.385 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:26:19.669 7f1edf75f088 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:26:19.669 7f1edf75f088 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:26:19.669 7f1edf75f088 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:26:19.673 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:26:19.673 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:26:19.674 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:26:19.674 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:26:19.675 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:26:19.675 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:26:19.679 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:26:19.679 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:26:19.680 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:26:19.680 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:26:19.680 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:26:19.680 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:26:19.680 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:26:19.680 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:26:19.680 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:26:19.680 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:26:19.680 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:26:19.680 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:26:19.681 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:26:19.682 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:26:19.682 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:26:19.682 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:26:19.682 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:26:19.683 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:26:19.683 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:26:19.683 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:26:19.684 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:26:19.684 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:26:19.684 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:26:19.684 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:26:19.684 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:26:19.684 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:26:19.685 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:26:19.685 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:26:19.686 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:26:19.686 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:26:19.686 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:26:19.686 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:26:19.686 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:26:19.686 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:26:19.687 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:26:19.687 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:26:19.687 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:26:19.687 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:26:19.687 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:26:19.687 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:26:19.687 7f1edf75f088 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:26:19.690 7f1edf75f088 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:26:19.737 7f1ed625ecc0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:26:19.737 7f1ed625ecc0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:26:19.737 7f1ed5a41438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:19.738 7f1ed5a41438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:26:19.738 7f1ed5a41438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:19.759 7f1ed5a0f4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:19.759 7f1ed5a0f4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:26:19.759 7f1ed5a0f4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:26:20.388 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:20.403 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:26:20.420 7f1edc6b3780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:26:20.439 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:26:21.444 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:22.057 7f1ed58ecc18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:26:22.065 7f1ed58ecc18 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:26:22.065 7f1ed58ecc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:26:22.065 7f1ed58ecc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:22.066 7f1ed58ecc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:26:22.066 7f1ed58ecc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:26:22.066 7f1ed58ecc18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:26:22.066 7f1ed58ecc18 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:26:22.066 7f1ed58ecc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:26:22.067 7f1ed58ecc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:26:22.067 7f1ed58ecc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:26:22.067 7f1ed58f3240 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:26:22.068 7f1ed58f3240 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:26:22.070 7f1ed58eca20 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:26:22.071 7f1ed58eca20 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:26:22.072 7f1ed7975e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:26:22.072 7f1ed7975e10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:26:22.072 7f1ed7975e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:26:22.072 7f1ed7975e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:26:22.072 7f1ed7975e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:26:22.073 7f1ed58f3a20 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:26:22.074 7f1ed58f3a20 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:26:22.445 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:23.446 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:24.447 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:25.448 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:26.448 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:27.449 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:28.450 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:29.451 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:30.453 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:31.454 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:32.455 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:33.456 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:34.457 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:35.458 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:36.459 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:37.460 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:38.462 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:39.463 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:40.465 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:41.466 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:42.467 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:43.468 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:44.469 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:45.470 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:46.471 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:47.472 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:48.473 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:49.474 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:50.475 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:51.477 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:52.478 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:53.479 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:54.479 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:55.480 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:56.481 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:57.482 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:58.483 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:26:59.484 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:00.485 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:01.486 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:02.488 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:03.490 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:04.491 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:05.492 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:06.494 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:07.495 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:08.496 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:09.497 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:10.500 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:11.500 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:12.502 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:13.503 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:14.505 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:15.506 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:16.507 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:17.508 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:18.510 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:19.511 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:20.512 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:27:20.907 7f1edf75f088 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:27:21.063 7f1edf75f088 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:27:21.550 7f1edc6b3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:04.070 7f7ccd9147c8 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:28:04.070 7f7ccd9147c8 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:28:04.071 7f7ccd9147c8 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:28:04.071 7f7cc4fac6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:28:04.071 7f7cc4fac6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:28:04.072 7f7cc4fac6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:28:04.072 7f7cc4fac6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:28:04.072 7f7cc4fac6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.072 7f7cc4fac6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:28:04.072 7f7cc4fac6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.072 7f7cc4fac6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.072 7f7cc4fac6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730946484071995 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:28:04.072 7f7cc4fac6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.108 7f7ccd9147c8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:28:04.113 7f7ccd9147c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:28:04.114 7f7cc4efc6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.114 7f7cc4efc6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:28:04.114 7f7cc4efc6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.114 7f7cc4efc6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.114 7f7cc4efc6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:28:04.114 7f7cc4efc6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.128 7f7ccd9147c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:28:04.128 7f7cc4ef5048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.129 7f7cc4ef5048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:28:04.129 7f7cc4ef5048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.136 7f7ccd9147c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:28:04.137 7f7cc4e382e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.137 7f7cc4e382e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:28:04.137 7f7cc4e382e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.151 7f7ccd9147c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:28:04.152 7f7cc4e38978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.152 7f7cc4e38978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:28:04.152 7f7cc4e38978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.161 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:28:04.161 7f7ccd9147c8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:28:04.162 7f7ccd9147c8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:28:04.162 7f7ccd9147c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:28:04.162 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:28:04.163 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:28:04.163 7f7cc4e27a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:28:04.163 7f7cc4e27a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:28:04.164 7f7cc4e38d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.164 7f7cc4e38d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:28:04.164 7f7cc4e38d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.165 7f7ccd9147c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:28:04.184 7f7cca89a780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:28:04.184 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:28:04.184 7f7cc4e27a20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:28:04.223 7f7ccd9147c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:28:04.225 7f7cc4d35d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.226 7f7cc4d35d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:28:04.226 7f7cc4d35d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.231 7f7ccd9147c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:28:04.231 7f7cc4d8c978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.231 7f7cc4d8c978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:28:04.231 7f7cc4d8c978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.243 7f7ccd9147c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:28:04.284 7f7ccd9147c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:28:04.285 7f7cc4d7b0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.285 7f7cc4d7b0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:28:04.285 7f7cc4d7b0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.297 7f7cc4d7b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.297 7f7cc4d7b8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:28:04.297 7f7cc4d7b8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.305 7f7cc4d828d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.306 7f7cc4d828d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:28:04.306 7f7cc4d828d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:04.306 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:28:04.306 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:28:04.306 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:28:04.309 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:28:04.323 7f7cca89a780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:28:04.331 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:28:04.343 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:28:04.357 7f7cca89a780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:28:04.365 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:28:04.822 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:28:04.992 7f7ccd9147c8 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:28:05.183 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:28:05.184 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:28:05.368 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:05.376 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:28:05.392 7f7cca89a780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:28:05.406 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:28:05.686 7f7ccd9147c8 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:28:05.686 7f7ccd9147c8 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:28:05.686 7f7ccd9147c8 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:28:05.690 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:28:05.690 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:28:05.691 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:28:05.691 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:28:05.692 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:28:05.692 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:28:05.697 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:28:05.697 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:28:05.697 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:28:05.697 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:28:05.697 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:28:05.697 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:28:05.697 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:28:05.697 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:28:05.698 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:28:05.698 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:28:05.698 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:28:05.698 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:28:05.698 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:28:05.699 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:28:05.700 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:28:05.700 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:28:05.700 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:28:05.700 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:28:05.700 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:28:05.700 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:28:05.701 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:28:05.701 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:28:05.701 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:28:05.701 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:28:05.701 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:28:05.701 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:28:05.702 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:28:05.702 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:28:05.702 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:28:05.702 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:28:05.702 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:28:05.703 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:28:05.703 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:28:05.703 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:28:05.703 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:28:05.703 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:28:05.703 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:28:05.703 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:28:05.703 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:28:05.703 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:28:05.703 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:28:05.708 7f7ccd9147c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:28:05.754 7f7cc4bf4390 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:28:05.754 7f7cc4bf4390 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:28:05.754 7f7cc43af198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:05.755 7f7cc43af198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:28:05.755 7f7cc43af198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:05.778 7f7cc4406240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:05.778 7f7cc4406240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:28:05.778 7f7cc4406240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:28:06.410 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:06.421 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:28:06.435 7f7cca89a780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:28:06.455 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:28:07.459 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:08.072 7f7cc42d86d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:28:08.079 7f7cc42d86d8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:28:08.079 7f7cc42d86d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:28:08.079 7f7cc42d86d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:08.079 7f7cc42d86d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:28:08.080 7f7cc42d86d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:28:08.080 7f7cc42d86d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:28:08.080 7f7cc42d86d8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:28:08.080 7f7cc42d86d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:28:08.080 7f7cc42d86d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:28:08.081 7f7cc42d86d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:28:08.081 7f7cc42eb198 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:28:08.082 7f7cc42eb198 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:28:08.086 7f7cc42eb048 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:28:08.087 7f7cc42eb048 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:28:08.088 7f7cc5b5de10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:28:08.088 7f7cc5b5de10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:28:08.088 7f7cc5b5de10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:28:08.088 7f7cc5b5de10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:28:08.089 7f7cc5b5de10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:28:08.089 7f7cc42d8f60 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:28:08.091 7f7cc42d8f60 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:28:08.461 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:09.462 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:10.463 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:11.464 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:12.466 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:13.467 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:14.469 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:15.470 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:16.471 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:17.472 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:18.473 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:19.474 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:20.475 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:21.476 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:22.477 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:23.478 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:24.480 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:25.481 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:26.482 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:27.483 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:28.485 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:29.486 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:30.487 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:31.488 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:32.489 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:33.490 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:34.490 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:35.491 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:36.493 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:37.495 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:38.495 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:39.496 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:40.497 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:41.499 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:42.500 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:43.502 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:44.502 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:45.504 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:46.505 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:47.506 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:48.508 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:49.509 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:50.510 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:51.511 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:52.512 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:53.513 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:54.515 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:55.515 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:56.516 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:57.518 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:58.519 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:28:59.520 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:00.521 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:01.522 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:02.523 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:03.525 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:04.526 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:05.527 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:06.529 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:07.059 7f7ccd9147c8 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:29:07.211 7f7ccd9147c8 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:29:07.659 7f7cca89a780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:50.012 7f9fe3bb4788 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:29:50.012 7f9fe3bb4788 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:29:50.012 7f9fe3bb4788 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:29:50.013 7f9fdb2846d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:29:50.013 7f9fdb2846d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:29:50.013 7f9fdb2846d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:29:50.014 7f9fdb2846d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:29:50.014 7f9fdb2846d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.014 7f9fdb2846d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:50.014 7f9fdb2846d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.014 7f9fdb2846d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.014 7f9fdb2846d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730946590013695 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:29:50.014 7f9fdb2846d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.053 7f9fe3bb4788 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:29:50.059 7f9fe3bb4788 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:29:50.060 7f9fdb1d46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.060 7f9fdb1d46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:29:50.061 7f9fdb1d46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.061 7f9fdb1d46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.061 7f9fdb1d46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:29:50.061 7f9fdb1d46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.075 7f9fe3bb4788 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:29:50.076 7f9fdb1cd048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.076 7f9fdb1cd048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:50.076 7f9fdb1cd048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.084 7f9fe3bb4788 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:29:50.085 7f9fdb1102e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.085 7f9fdb1102e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:29:50.085 7f9fdb1102e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.100 7f9fe3bb4788 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:29:50.101 7f9fdb110978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.101 7f9fdb110978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:50.101 7f9fdb110978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.112 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:29:50.113 7f9fe3bb4788 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:29:50.114 7f9fe3bb4788 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:29:50.114 7f9fe3bb4788 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:29:50.114 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:29:50.115 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:50.115 7f9fdb10ea20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:29:50.115 7f9fdb10ea20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:29:50.116 7f9fdb110d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.116 7f9fdb110d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:29:50.116 7f9fdb110d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.118 7f9fe3bb4788 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:29:50.137 7f9fdb10ea20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:29:50.138 7f9fe0b72780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:50.138 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:29:50.178 7f9fe3bb4788 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:29:50.183 7f9fda00e8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.183 7f9fda00e8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:29:50.183 7f9fda00e8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.188 7f9fe3bb4788 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:29:50.189 7f9fda010eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.189 7f9fda010eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:29:50.189 7f9fda010eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.201 7f9fe3bb4788 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:29:50.242 7f9fe3bb4788 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:29:50.243 7f9fda051630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.243 7f9fda051630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:50.243 7f9fda051630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.256 7f9fda051e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.256 7f9fda051e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:29:50.256 7f9fda051e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.265 7f9fda05de10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.265 7f9fda05de10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:50.266 7f9fda05de10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:50.266 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:29:50.266 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:29:50.266 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:50.267 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:29:50.282 7f9fe0b72780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:50.291 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:29:50.299 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:50.314 7f9fe0b72780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:50.321 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:29:50.780 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:29:50.966 7f9fe3bb4788 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:29:51.150 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:29:51.151 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:29:51.324 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:51.331 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:51.345 7f9fe0b72780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:51.354 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:29:51.652 7f9fe3bb4788 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:29:51.653 7f9fe3bb4788 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:29:51.653 7f9fe3bb4788 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:29:51.656 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:29:51.656 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:29:51.657 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:29:51.657 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:29:51.658 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:29:51.659 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:29:51.663 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:29:51.663 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:29:51.663 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:29:51.663 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:29:51.663 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:29:51.663 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:29:51.664 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:29:51.664 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:29:51.664 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:29:51.664 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:29:51.664 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:29:51.664 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:29:51.664 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:29:51.665 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:29:51.666 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:29:51.666 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:29:51.666 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:29:51.666 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:29:51.666 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:29:51.666 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:29:51.667 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:29:51.667 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:29:51.667 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:29:51.667 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:29:51.667 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:29:51.667 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:29:51.668 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:29:51.668 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:29:51.669 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:29:51.669 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:29:51.669 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:29:51.669 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:29:51.669 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:29:51.669 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:29:51.669 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:29:51.669 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:29:51.670 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:29:51.670 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:29:51.670 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:29:51.670 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:29:51.670 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:29:51.673 7f9fe3bb4788 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:29:51.721 7f9fd9dfc6d8 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:29:51.722 7f9fd9dfc6d8 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:29:51.722 7f9fd96f46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:51.722 7f9fd96f46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:29:51.722 7f9fd96f46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:51.743 7f9fd96d1780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:51.744 7f9fd96d1780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:29:51.744 7f9fd96d1780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:29:52.358 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:52.367 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:29:52.382 7f9fe0b72780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:29:52.400 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:29:53.404 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:54.013 7f9fd95abeb8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:29:54.020 7f9fd95abeb8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:29:54.020 7f9fd95abeb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:29:54.020 7f9fd95abeb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:29:54.021 7f9fd95abeb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:29:54.021 7f9fd95abeb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:29:54.021 7f9fd95abeb8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:29:54.021 7f9fd95abeb8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:29:54.021 7f9fd95abeb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:29:54.021 7f9fd95abeb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:29:54.022 7f9fd95abeb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:29:54.022 7f9fd95456d8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:29:54.023 7f9fd95456d8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:29:54.026 7f9fd9545588 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:29:54.026 7f9fd9545588 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:29:54.027 7f9fdbe9fd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:29:54.027 7f9fdbe9fd68 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:29:54.027 7f9fdbe9fd68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:29:54.027 7f9fdbe9fd68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:29:54.027 7f9fdbe9fd68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:29:54.028 7f9fd95452e8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:29:54.029 7f9fd95452e8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:29:54.405 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:55.406 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:56.407 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:57.409 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:58.410 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:29:59.411 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:00.412 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:01.413 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:02.414 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:03.415 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:04.417 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:05.417 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:06.419 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:07.420 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:08.421 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:09.423 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:10.424 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:11.426 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:12.426 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:13.427 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:14.428 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:15.159 7f9fe0d4cc18 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:30:15.429 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:16.430 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:17.431 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:18.433 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:19.434 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:20.436 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:21.437 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:22.438 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:23.439 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:24.440 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:25.441 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:26.443 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:27.444 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:28.445 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:29.446 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:30.447 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:31.449 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:32.450 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:33.450 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:34.451 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:35.453 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:36.454 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:37.456 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:38.457 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:39.458 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:40.459 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:41.459 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:42.461 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:43.462 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:44.464 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:45.464 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:46.466 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:47.467 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:48.468 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:49.469 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:50.470 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:51.471 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:52.472 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:30:53.303 7f9fe3bb4788 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:30:53.463 7f9fe3bb4788 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:30:53.473 7f9fe0b72780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:36.076 7f194bbc0c08 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:31:36.076 7f194bbc0c08 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:31:36.076 7f194bbc0c08 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:31:36.076 7f19431f46d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:31:36.076 7f19431f46d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:31:36.077 7f19431f46d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:31:36.077 7f19431f46d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:31:36.077 7f19431f46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.077 7f19431f46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:36.078 7f19431f46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.078 7f19431f46d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.078 7f19431f46d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730946696077147 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:31:36.078 7f19431f46d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.120 7f194bbc0c08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:31:36.125 7f194bbc0c08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:31:36.126 7f19431446d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.126 7f19431446d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:31:36.126 7f19431446d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.126 7f19431446d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.127 7f19431446d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:31:36.127 7f19431446d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.140 7f194bbc0c08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:31:36.141 7f194313d048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.141 7f194313d048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:36.141 7f194313d048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.149 7f194bbc0c08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:31:36.150 7f19430802e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.150 7f19430802e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:31:36.150 7f19430802e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.162 7f194bbc0c08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:31:36.163 7f1943080978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.163 7f1943080978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:36.163 7f1943080978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.170 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:31:36.170 7f194bbc0c08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:31:36.170 7f194bbc0c08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:31:36.171 7f194bbc0c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:31:36.171 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:31:36.171 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:31:36.172 7f194307fa20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:31:36.172 7f194307fa20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:31:36.172 7f1943080d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.172 7f1943080d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:31:36.173 7f1943080d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.174 7f194bbc0c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:31:36.188 7f194307fa20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:31:36.189 7f1948ae3780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:31:36.189 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:31:36.228 7f194bbc0c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:31:36.230 7f194277db70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.230 7f194277db70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:31:36.230 7f194277db70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.235 7f194bbc0c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:31:36.235 7f1942819c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.235 7f1942819c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:31:36.235 7f1942819c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.247 7f194bbc0c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:31:36.284 7f194bbc0c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:31:36.285 7f194277f390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.286 7f194277f390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:36.286 7f194277f390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.298 7f194277fb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.298 7f194277fb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:31:36.298 7f194277fb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.308 7f19427c9b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.308 7f19427c9b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:36.308 7f19427c9b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:36.308 7f194bbc0c08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:31:36.309 7f194bbc0c08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:31:36.309 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:31:36.310 7f194bbc0c08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:31:36.323 7f1948ae3780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:31:36.331 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:31:36.339 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:31:36.353 7f1948ae3780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:31:36.360 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:31:36.823 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:31:37.003 7f194bbc0c08 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:31:37.182 7f194bbc0c08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:31:37.183 7f194bbc0c08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:31:37.362 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:37.369 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:31:37.393 7f1948ae3780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:31:37.407 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:31:37.684 7f194bbc0c08 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:31:37.684 7f194bbc0c08 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:31:37.684 7f194bbc0c08 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:31:37.688 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:31:37.688 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:31:37.689 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:31:37.689 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:31:37.690 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:31:37.690 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:31:37.695 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:31:37.695 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:31:37.695 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:31:37.695 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:31:37.695 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:31:37.695 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:31:37.695 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:31:37.695 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:31:37.695 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:31:37.695 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:31:37.696 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:31:37.696 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:31:37.696 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:31:37.697 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:31:37.697 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:31:37.697 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:31:37.697 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:31:37.698 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:31:37.698 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:31:37.698 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:31:37.699 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:31:37.699 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:31:37.699 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:31:37.699 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:31:37.699 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:31:37.699 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:31:37.700 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:31:37.700 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:31:37.700 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:31:37.700 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:31:37.700 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:31:37.700 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:31:37.700 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:31:37.700 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:31:37.701 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:31:37.701 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:31:37.701 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:31:37.701 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:31:37.701 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:31:37.701 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:31:37.701 7f194bbc0c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:31:37.704 7f194bbc0c08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:31:37.751 7f194268ff60 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:31:37.751 7f194268ff60 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:31:37.751 7f1941e73438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:37.752 7f1941e73438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:31:37.752 7f1941e73438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:37.773 7f1941e414e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:37.773 7f1941e414e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:31:37.774 7f1941e414e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:31:38.410 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:38.423 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:31:38.437 7f1948ae3780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:31:38.453 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:31:39.458 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:40.077 7f1941d1dc18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:31:40.084 7f1941d1dc18 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:31:40.085 7f1941d1dc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:31:40.085 7f1941d1dc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:31:40.085 7f1941d1dc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:31:40.085 7f1941d1dc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:31:40.085 7f1941d1dc18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:31:40.086 7f1941d1dc18 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:31:40.086 7f1941d1dc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:31:40.086 7f1941d1dc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:31:40.086 7f1941d1dc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:31:40.087 7f1941d35438 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:31:40.088 7f1941d35438 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:31:40.092 7f1941d352e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:31:40.093 7f1941d352e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:31:40.093 7f1943da6e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:31:40.093 7f1943da6e10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:31:40.093 7f1943da6e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:31:40.094 7f1943da6e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:31:40.094 7f1943da6e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:31:40.094 7f1941d35048 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:31:40.095 7f1941d35048 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:31:40.458 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:41.459 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:42.461 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:43.462 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:44.463 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:45.465 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:46.467 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:47.468 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:48.469 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:49.470 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:50.471 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:51.473 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:52.475 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:53.476 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:54.478 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:55.479 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:56.480 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:57.481 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:58.483 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:31:59.485 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:00.485 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:01.486 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:02.487 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:03.488 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:04.489 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:05.491 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:06.492 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:07.493 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:08.494 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:09.495 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:10.496 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:11.498 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:12.499 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:13.500 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:14.501 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:15.502 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:16.503 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:17.504 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:18.506 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:19.507 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:20.508 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:21.510 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:22.511 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:23.512 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:24.513 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:25.515 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:26.516 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:27.517 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:28.518 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:29.519 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:30.520 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:31.522 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:32.523 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:33.525 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:34.526 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:35.527 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:36.528 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:37.529 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:38.530 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:32:39.083 7f194bbc0c08 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:32:39.247 7f194bbc0c08 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:32:39.532 7f1948ae3780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:22.038 7fe49ca60b88 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:33:22.038 7fe49ca60b88 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:33:22.038 7fe49ca60b88 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:33:22.038 7fe49412b6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:33:22.038 7fe49412b6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:33:22.039 7fe49412b6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:33:22.039 7fe49412b6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:33:22.039 7fe49412b6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.040 7fe49412b6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:33:22.040 7fe49412b6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.040 7fe49412b6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.040 7fe49412b6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730946802039280 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:33:22.040 7fe49412b6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.080 7fe49ca60b88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:33:22.085 7fe49ca60b88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:33:22.086 7fe49407f6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.086 7fe49407f6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:33:22.086 7fe49407f6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.086 7fe49407f6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.086 7fe49407f6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:33:22.086 7fe49407f6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.099 7fe49ca60b88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:33:22.099 7fe494079048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.099 7fe494079048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:33:22.100 7fe494079048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.107 7fe49ca60b88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:33:22.108 7fe48ffb72e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.108 7fe48ffb72e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:33:22.108 7fe48ffb72e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.121 7fe49ca60b88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:33:22.122 7fe48ffb7780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.122 7fe48ffb7780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:33:22.122 7fe48ffb7780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.130 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:33:22.130 7fe49ca60b88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:33:22.131 7fe49ca60b88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:33:22.131 7fe49ca60b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:33:22.131 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:33:22.131 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:33:22.132 7fe49c96d438 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:33:22.132 7fe49c96d438 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:33:22.133 7fe48ffb7d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.133 7fe48ffb7d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:33:22.133 7fe48ffb7d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.134 7fe49ca60b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:33:22.154 7fe48ffb5f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:33:22.154 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:33:22.154 7fe49c96d438 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:33:22.192 7fe49ca60b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:33:22.194 7fe48f6b4b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.194 7fe48f6b4b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:33:22.194 7fe48f6b4b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.199 7fe49ca60b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:33:22.200 7fe48f752c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.200 7fe48f752c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:33:22.200 7fe48f752c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.212 7fe49ca60b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:33:22.256 7fe49ca60b88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:33:22.257 7fe48f6b8390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.257 7fe48f6b8390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:33:22.257 7fe48f6b8390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.271 7fe48f6b8b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.271 7fe48f6b8b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:33:22.271 7fe48f6b8b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.280 7fe48f6ffb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.280 7fe48f6ffb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:33:22.280 7fe48f6ffb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:22.281 7fe49ca60b88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:33:22.281 7fe49ca60b88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:33:22.281 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:33:22.282 7fe49ca60b88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:33:22.295 7fe48ffb5f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:33:22.304 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:33:22.312 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:33:22.325 7fe48ffb5f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:33:22.332 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:33:22.795 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:33:22.966 7fe49ca60b88 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:33:23.147 7fe49ca60b88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:33:23.148 7fe49ca60b88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:33:23.335 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:23.343 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:33:23.357 7fe48ffb5f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:33:23.368 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:33:23.650 7fe49ca60b88 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:33:23.650 7fe49ca60b88 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:33:23.650 7fe49ca60b88 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:33:23.654 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:33:23.654 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:33:23.655 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:33:23.655 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:33:23.656 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:33:23.656 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:33:23.662 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:33:23.662 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:33:23.663 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:33:23.663 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:33:23.663 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:33:23.663 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:33:23.663 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:33:23.663 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:33:23.663 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:33:23.664 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:33:23.664 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:33:23.664 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:33:23.664 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:33:23.666 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:33:23.666 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:33:23.666 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:33:23.666 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:33:23.667 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:33:23.667 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:33:23.667 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:33:23.668 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:33:23.668 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:33:23.668 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:33:23.668 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:33:23.668 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:33:23.668 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:33:23.669 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:33:23.669 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:33:23.669 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:33:23.670 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:33:23.670 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:33:23.670 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:33:23.670 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:33:23.670 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:33:23.670 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:33:23.670 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:33:23.671 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:33:23.671 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:33:23.671 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:33:23.671 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:33:23.671 7fe49ca60b88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:33:23.674 7fe49ca60b88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:33:23.725 7fe48f5c6198 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:33:23.726 7fe48f5c6198 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:33:23.726 7fe48eda9438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:23.726 7fe48eda9438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:33:23.726 7fe48eda9438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:23.749 7fe48ed774e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:23.749 7fe48ed774e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:33:23.749 7fe48ed774e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:33:24.371 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:24.383 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:33:24.398 7fe48ffb5f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:33:24.413 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:33:25.417 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:26.039 7fe48ec5b4e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:33:26.046 7fe48ec5b4e0 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:33:26.047 7fe48ec5b4e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:33:26.047 7fe48ec5b4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:33:26.048 7fe48ec5b4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:33:26.048 7fe48ec5b4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:33:26.048 7fe48ec5b4e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:33:26.048 7fe48ec5b4e0 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:33:26.049 7fe48ec5b4e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:33:26.049 7fe48ec5b4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:33:26.049 7fe48ec5b4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:33:26.049 7fe48ebed438 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:33:26.051 7fe48ebed438 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:33:26.055 7fe48ebed2e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:33:26.056 7fe48ebed2e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:33:26.056 7fe494d49d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:33:26.056 7fe494d49d68 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:33:26.056 7fe494d49d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:33:26.056 7fe494d49d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:33:26.057 7fe494d49d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:33:26.057 7fe48ec5b4e0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:33:26.058 7fe48ec5b4e0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:33:26.419 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:27.420 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:28.421 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:29.423 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:30.424 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:31.426 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:32.428 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:33.429 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:34.431 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:35.431 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:36.433 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:37.433 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:38.435 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:39.435 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:40.437 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:41.438 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:42.439 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:43.440 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:44.441 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:45.442 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:46.443 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:47.443 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:48.444 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:49.446 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:50.447 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:51.448 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:52.448 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:53.449 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:54.450 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:55.452 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:56.454 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:57.455 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:58.456 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:33:59.458 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:00.458 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:01.459 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:02.461 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:03.462 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:04.463 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:05.465 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:06.466 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:07.467 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:08.469 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:09.471 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:10.472 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:11.473 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:12.474 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:13.475 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:14.476 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:15.477 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:16.479 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:17.480 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:18.483 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:19.485 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:20.487 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:21.492 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:22.492 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:23.494 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:24.496 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:25.497 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:26.498 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:27.499 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:34:27.835 7fe49ca60b88 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:34:28.003 7fe49ca60b88 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:34:28.531 7fe48ffb5f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:08.035 7fd9dd2ae808 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:35:08.035 7fd9dd2ae808 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:35:08.035 7fd9dd2ae808 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:35:08.036 7fd9d494b6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:35:08.036 7fd9d494b6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:35:08.036 7fd9d494b6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:35:08.037 7fd9d494b6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:35:08.037 7fd9d494b6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.038 7fd9d494b6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:35:08.038 7fd9d494b6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.038 7fd9d494b6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.039 7fd9d494b6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730946908036848 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:35:08.039 7fd9d494b6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.077 7fd9dd2ae808 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:35:08.082 7fd9dd2ae808 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:35:08.082 7fd9d489e6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.082 7fd9d489e6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:35:08.083 7fd9d489e6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.083 7fd9d489e6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.083 7fd9d489e6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:35:08.083 7fd9d489e6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.100 7fd9dd2ae808 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:35:08.100 7fd9d4898048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.100 7fd9d4898048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:35:08.101 7fd9d4898048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.109 7fd9dd2ae808 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:35:08.110 7fd9d47da2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.110 7fd9d47da2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:35:08.110 7fd9d47da2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.125 7fd9dd2ae808 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:35:08.125 7fd9d47da780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.125 7fd9d47da780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:35:08.125 7fd9d47da780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.133 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:35:08.134 7fd9dd2ae808 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:35:08.134 7fd9dd2ae808 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:35:08.135 7fd9dd2ae808 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:35:08.135 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:35:08.135 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:35:08.136 7fd9dd1864e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:35:08.136 7fd9dd1864e0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:35:08.136 7fd9d47dad68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.136 7fd9d47dad68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:35:08.136 7fd9d47dad68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.137 7fd9dd2ae808 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:35:08.154 7fd9dd1864e0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:35:08.154 7fd9d47d8f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:35:08.155 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:35:08.194 7fd9dd2ae808 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:35:08.196 7fd9d46d5d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.196 7fd9d46d5d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:35:08.197 7fd9d46d5d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.201 7fd9dd2ae808 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:35:08.201 7fd9d472b978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.202 7fd9d472b978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:35:08.202 7fd9d472b978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.214 7fd9dd2ae808 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:35:08.255 7fd9dd2ae808 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:35:08.256 7fd9d46c40f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.256 7fd9d46c40f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:35:08.256 7fd9d46c40f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.267 7fd9d46c48d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.268 7fd9d46c48d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:35:08.268 7fd9d46c48d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.276 7fd9d47258d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.276 7fd9d47258d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:35:08.276 7fd9d47258d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:08.277 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:35:08.277 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:35:08.277 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:35:08.278 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:35:08.292 7fd9d47d8f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:35:08.300 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:35:08.308 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:35:08.322 7fd9d47d8f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:35:08.330 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:35:08.793 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:35:08.976 7fd9dd2ae808 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:35:09.179 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:35:09.180 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:35:09.333 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:09.342 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:35:09.358 7fd9d47d8f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:35:09.370 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:35:09.681 7fd9dd2ae808 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:35:09.682 7fd9dd2ae808 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:35:09.682 7fd9dd2ae808 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:35:09.686 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:35:09.687 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:35:09.688 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:35:09.688 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:35:09.689 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:35:09.689 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:35:09.694 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:35:09.694 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:35:09.694 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:35:09.694 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:35:09.695 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:35:09.695 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:35:09.695 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:35:09.695 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:35:09.695 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:35:09.695 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:35:09.695 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:35:09.695 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:35:09.695 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:35:09.697 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:35:09.697 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:35:09.698 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:35:09.698 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:35:09.698 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:35:09.699 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:35:09.699 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:35:09.699 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:35:09.699 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:35:09.699 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:35:09.700 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:35:09.700 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:35:09.700 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:35:09.701 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:35:09.701 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:35:09.701 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:35:09.702 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:35:09.702 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:35:09.702 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:35:09.702 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:35:09.702 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:35:09.703 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:35:09.703 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:35:09.703 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:35:09.703 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:35:09.703 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:35:09.703 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:35:09.703 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:35:09.707 7fd9dd2ae808 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:35:09.767 7fd9d4595390 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:35:09.767 7fd9d4595390 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:35:09.767 7fd9d424a198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:09.768 7fd9d424a198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:35:09.768 7fd9d424a198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:09.802 7fd9d42a0240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:09.802 7fd9d42a0240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:35:09.803 7fd9d42a0240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:35:10.374 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:10.381 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:35:10.396 7fd9d47d8f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:35:10.424 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:35:11.433 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:12.037 7fd9d41716d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:35:12.049 7fd9d41716d8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:35:12.049 7fd9d41716d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:35:12.050 7fd9d41716d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:35:12.051 7fd9d41716d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:35:12.051 7fd9d41716d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:35:12.051 7fd9d41716d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:35:12.051 7fd9d41716d8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:35:12.052 7fd9d41716d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:35:12.052 7fd9d41716d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:35:12.052 7fd9d41716d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:35:12.052 7fd9d4186198 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:35:12.054 7fd9d4186198 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:35:12.057 7fd9d4186048 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:35:12.057 7fd9d4186048 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:35:12.058 7fd9d5568d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:35:12.058 7fd9d5568d68 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:35:12.058 7fd9d5568d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:35:12.059 7fd9d5568d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:35:12.059 7fd9d5568d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:35:12.060 7fd9d41716d8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:35:12.063 7fd9d41716d8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:35:12.434 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:13.435 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:14.435 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:15.436 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:16.437 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:17.438 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:18.444 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:19.445 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:20.446 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:21.447 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:22.448 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:23.449 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:24.450 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:25.451 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:26.452 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:27.453 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:28.454 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:29.455 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:30.456 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:31.456 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:32.457 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:33.459 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:34.459 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:35.461 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:36.462 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:37.463 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:38.464 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:39.465 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:40.467 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:41.468 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:42.469 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:43.471 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:44.472 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:45.472 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:46.473 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:46.707 7fd9d4176c18 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:35:47.474 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:48.475 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:49.476 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:50.478 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:51.480 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:52.481 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:52.494 7fd9dd36ceb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:35:53.482 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:54.483 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:55.484 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:56.485 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:57.486 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:58.487 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:35:59.488 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:00.489 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:01.490 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:02.492 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:03.493 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:04.494 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:05.495 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:06.497 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:07.498 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:08.500 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:09.501 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:10.503 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:11.151 7fd9dd2ae808 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:36:11.315 7fd9dd2ae808 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:36:11.504 7fd9d47d8f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:56.039 7f72d2c3d648 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:36:56.039 7f72d2c3d648 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:36:56.039 7f72d2c3d648 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:36:56.039 7f72ca3046d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:36:56.040 7f72ca3046d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:36:56.040 7f72ca3046d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:36:56.040 7f72ca3046d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:36:56.040 7f72ca3046d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.041 7f72ca3046d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:36:56.041 7f72ca3046d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.041 7f72ca3046d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.041 7f72ca3046d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730947016040438 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:36:56.041 7f72ca3046d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.078 7f72d2c3d648 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:36:56.083 7f72d2c3d648 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:36:56.083 7f72ca2556d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.083 7f72ca2556d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:36:56.083 7f72ca2556d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.084 7f72ca2556d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.084 7f72ca2556d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:36:56.084 7f72ca2556d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.096 7f72d2c3d648 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:36:56.097 7f72ca24f048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.097 7f72ca24f048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:36:56.097 7f72ca24f048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.104 7f72d2c3d648 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:36:56.104 7f72ca1902e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.104 7f72ca1902e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:36:56.104 7f72ca1902e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.116 7f72d2c3d648 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:36:56.117 7f72ca190780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.117 7f72ca190780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:36:56.117 7f72ca190780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.127 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:36:56.127 7f72d2c3d648 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:36:56.127 7f72d2c3d648 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:36:56.128 7f72d2c3d648 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:36:56.128 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:36:56.128 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:36:56.129 7f72d28dd630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:36:56.129 7f72d28dd630 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:36:56.129 7f72ca190d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.129 7f72ca190d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:36:56.130 7f72ca190d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.131 7f72d2c3d648 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:36:56.146 7f72ca18ff60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:36:56.147 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:36:56.147 7f72d28dd630 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:36:56.188 7f72d2c3d648 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:36:56.190 7f72c988eb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.191 7f72c988eb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:36:56.191 7f72c988eb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.195 7f72d2c3d648 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:36:56.196 7f72c98f4c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.196 7f72c98f4c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:36:56.196 7f72c98f4c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.207 7f72d2c3d648 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:36:56.246 7f72d2c3d648 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:36:56.247 7f72c9890390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.247 7f72c9890390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:36:56.247 7f72c9890390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.258 7f72c9890b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.259 7f72c9890b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:36:56.259 7f72c9890b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.267 7f72c98d9b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.267 7f72c98d9b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:36:56.267 7f72c98d9b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:56.268 7f72d2c3d648 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:36:56.268 7f72d2c3d648 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:36:56.268 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:36:56.269 7f72d2c3d648 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:36:56.281 7f72ca18ff60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:36:56.288 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:36:56.299 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:36:56.313 7f72ca18ff60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:36:56.320 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:36:56.782 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:36:56.956 7f72d2c3d648 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:36:57.157 7f72d2c3d648 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:36:57.163 7f72d2c3d648 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:36:57.322 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:57.330 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:36:57.345 7f72ca18ff60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:36:57.358 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:36:57.664 7f72d2c3d648 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:36:57.664 7f72d2c3d648 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:36:57.664 7f72d2c3d648 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:36:57.667 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:36:57.667 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:36:57.669 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:36:57.669 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:36:57.670 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:36:57.670 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:36:57.675 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:36:57.675 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:36:57.675 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:36:57.675 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:36:57.675 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:36:57.675 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:36:57.675 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:36:57.675 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:36:57.675 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:36:57.676 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:36:57.676 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:36:57.676 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:36:57.676 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:36:57.677 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:36:57.677 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:36:57.678 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:36:57.678 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:36:57.678 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:36:57.678 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:36:57.678 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:36:57.679 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:36:57.679 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:36:57.680 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:36:57.680 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:36:57.680 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:36:57.680 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:36:57.681 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:36:57.681 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:36:57.681 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:36:57.681 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:36:57.681 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:36:57.681 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:36:57.682 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:36:57.682 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:36:57.682 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:36:57.682 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:36:57.682 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:36:57.682 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:36:57.682 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:36:57.682 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:36:57.682 7f72d2c3d648 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:36:57.685 7f72d2c3d648 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:36:57.732 7f72c979e8d0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:36:57.733 7f72c979e8d0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:36:57.733 7f72c8f83438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:57.733 7f72c8f83438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:36:57.733 7f72c8f83438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:57.752 7f72c8f514e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:57.752 7f72c8f514e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:36:57.752 7f72c8f514e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:36:58.363 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:36:58.371 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:36:58.387 7f72ca18ff60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:36:58.403 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:36:59.406 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:00.041 7f72c8e66b70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:37:00.049 7f72c8e66b70 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:37:00.049 7f72c8e66b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:37:00.049 7f72c8e66b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:37:00.049 7f72c8e66b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:37:00.050 7f72c8e66b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:37:00.050 7f72c8e66b70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:37:00.050 7f72c8e66b70 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:37:00.051 7f72c8e66b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:37:00.051 7f72c8e66b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:37:00.051 7f72c8e66b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:37:00.051 7f72c8e45438 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:37:00.053 7f72c8e45438 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:37:00.057 7f72c8e452e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:37:00.057 7f72c8e452e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:37:00.058 7f72c8f514e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:37:00.058 7f72c8f514e0 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:37:00.058 7f72c8f514e0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:37:00.059 7f72c8f514e0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:37:00.059 7f72c8f514e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:37:00.059 7f72c8e66b70 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:37:00.061 7f72c8e66b70 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:37:00.408 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:01.409 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:02.411 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:03.411 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:04.414 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:05.415 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:06.416 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:07.416 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:08.417 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:09.418 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:10.420 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:11.421 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:12.422 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:13.423 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:14.424 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:15.426 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:16.427 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:17.429 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:18.430 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:19.430 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:20.432 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:21.433 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:22.434 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:23.435 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:24.436 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:25.437 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:26.438 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:27.440 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:28.441 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:29.442 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:30.445 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:31.446 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:32.448 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:33.449 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:34.450 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:35.451 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:36.452 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:37.453 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:38.455 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:39.457 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:40.458 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:41.458 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:42.460 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:43.461 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:44.462 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:45.463 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:46.465 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:47.466 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:48.467 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:49.468 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:50.469 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:51.470 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:52.471 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:53.473 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:54.474 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:55.475 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:56.476 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:57.477 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:58.478 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:37:58.991 7f72d2c3d648 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:37:59.147 7f72d2c3d648 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:37:59.479 7f72ca18ff60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:44.060 7f9e65cfbd48 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:38:44.060 7f9e65cfbd48 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:38:44.060 7f9e65cfbd48 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:38:44.061 7f9e5d38d6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:38:44.061 7f9e5d38d6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:38:44.062 7f9e5d38d6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:38:44.062 7f9e5d38d6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:38:44.062 7f9e5d38d6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.062 7f9e5d38d6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:38:44.062 7f9e5d38d6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.062 7f9e5d38d6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.062 7f9e5d38d6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730947124061944 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:38:44.062 7f9e5d38d6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.098 7f9e65cfbd48 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:38:44.103 7f9e65cfbd48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:38:44.104 7f9e5d2e26d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.104 7f9e5d2e26d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:38:44.104 7f9e5d2e26d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.104 7f9e5d2e26d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.104 7f9e5d2e26d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:38:44.104 7f9e5d2e26d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.118 7f9e65cfbd48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:38:44.118 7f9e5d2db048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.118 7f9e5d2db048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:38:44.118 7f9e5d2db048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.131 7f9e65cfbd48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:38:44.132 7f9e5d21d2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.132 7f9e5d21d2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:38:44.132 7f9e5d21d2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.144 7f9e65cfbd48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:38:44.145 7f9e5d21d780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.145 7f9e5d21d780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:38:44.145 7f9e5d21d780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.154 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:38:44.155 7f9e65cfbd48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:38:44.155 7f9e65cfbd48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:38:44.155 7f9e65cfbd48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:38:44.156 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:38:44.156 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:38:44.156 7f9e65bcc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:38:44.156 7f9e65bcc780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:38:44.157 7f9e5d21dd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.157 7f9e5d21dd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:38:44.157 7f9e5d21dd68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.158 7f9e65cfbd48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:38:44.175 7f9e65bcc780 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:38:44.176 7f9e5d21bf60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:38:44.176 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:38:44.221 7f9e65cfbd48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:38:44.223 7f9e5c91bb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.224 7f9e5c91bb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:38:44.224 7f9e5c91bb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.229 7f9e65cfbd48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:38:44.229 7f9e5c9a5c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.229 7f9e5c9a5c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:38:44.229 7f9e5c9a5c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.241 7f9e65cfbd48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:38:44.282 7f9e65cfbd48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:38:44.282 7f9e5c91e390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.283 7f9e5c91e390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:38:44.283 7f9e5c91e390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.295 7f9e5c91eb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.295 7f9e5c91eb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:38:44.295 7f9e5c91eb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.305 7f9e5c968b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.305 7f9e5c968b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:38:44.305 7f9e5c968b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:44.306 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:38:44.306 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:38:44.306 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:38:44.306 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:38:44.319 7f9e5d21bf60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:38:44.328 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:38:44.336 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:38:44.351 7f9e5d21bf60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:38:44.359 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:38:44.821 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:38:45.013 7f9e65cfbd48 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:38:45.200 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:38:45.200 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:38:45.362 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:45.369 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:38:45.383 7f9e5d21bf60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:38:45.395 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:38:45.701 7f9e65cfbd48 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:38:45.701 7f9e65cfbd48 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:38:45.701 7f9e65cfbd48 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:38:45.706 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:38:45.706 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:38:45.707 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:38:45.707 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:38:45.708 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:38:45.708 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:38:45.712 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:38:45.712 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:38:45.712 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:38:45.712 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:38:45.712 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:38:45.712 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:38:45.712 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:38:45.712 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:38:45.712 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:38:45.712 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:38:45.713 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:38:45.713 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:38:45.713 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:38:45.714 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:38:45.714 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:38:45.714 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:38:45.715 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:38:45.715 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:38:45.715 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:38:45.716 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:38:45.716 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:38:45.716 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:38:45.716 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:38:45.716 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:38:45.716 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:38:45.717 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:38:45.717 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:38:45.717 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:38:45.717 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:38:45.718 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:38:45.718 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:38:45.718 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:38:45.718 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:38:45.718 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:38:45.718 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:38:45.718 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:38:45.718 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:38:45.719 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:38:45.719 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:38:45.719 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:38:45.719 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:38:45.721 7f9e65cfbd48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:38:45.769 7f9e5c7d9b70 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:38:45.770 7f9e5c7d9b70 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:38:45.770 7f9e57ffc438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:45.770 7f9e57ffc438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:38:45.770 7f9e57ffc438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:45.791 7f9e57fca4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:45.792 7f9e57fca4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:38:45.792 7f9e57fca4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:38:46.397 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:46.407 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:38:46.422 7f9e5d21bf60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:38:46.441 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:38:47.446 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:48.062 7f9e57ea7c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:38:48.069 7f9e57ea7c18 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:38:48.069 7f9e57ea7c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:38:48.069 7f9e57ea7c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:38:48.069 7f9e57ea7c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:38:48.069 7f9e57ea7c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:38:48.069 7f9e57ea7c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:38:48.069 7f9e57ea7c18 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:38:48.070 7f9e57ea7c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:38:48.070 7f9e57ea7c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:38:48.070 7f9e57ea7c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:38:48.070 7f9e57ebf438 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:38:48.071 7f9e57ebf438 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:38:48.074 7f9e57ebf2e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:38:48.075 7f9e57ebf2e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:38:48.075 7f9e5df43e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:38:48.075 7f9e5df43e10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:38:48.075 7f9e5df43e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:38:48.076 7f9e5df43e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:38:48.076 7f9e5df43e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:38:48.076 7f9e57ebf048 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:38:48.077 7f9e57ebf048 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:38:48.447 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:49.448 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:50.449 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:51.451 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:52.453 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:53.454 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:54.456 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:55.456 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:56.457 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:57.458 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:58.460 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:38:59.461 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:00.463 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:01.464 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:02.465 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:03.466 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:04.466 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:05.467 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:06.468 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:07.469 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:08.470 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:09.472 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:10.473 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:11.473 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:12.474 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:13.475 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:14.477 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:15.478 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:16.480 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:17.481 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:18.482 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:19.484 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:20.485 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:21.488 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:22.488 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:23.500 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:24.506 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:25.507 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:26.512 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:27.525 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:28.525 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:29.526 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:30.530 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:31.533 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:32.541 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:33.542 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:34.542 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:35.543 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:36.545 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:37.547 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:38.547 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:39.548 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:40.549 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:41.551 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:42.553 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:43.554 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:44.555 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:45.556 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:45.596 7f9e5c9a5cc0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:39:46.557 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:47.558 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:48.560 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:49.561 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:39:50.383 7f9e65cfbd48 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:39:50.539 7f9e65cfbd48 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:39:50.562 7f9e5d21bf60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:30.044 7f6e81082cc8 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:40:30.044 7f6e81082cc8 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:40:30.044 7f6e81082cc8 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:40:30.045 7f6e787156d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:40:30.045 7f6e787156d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:40:30.046 7f6e787156d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:40:30.046 7f6e787156d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:40:30.046 7f6e787156d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.046 7f6e787156d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:40:30.046 7f6e787156d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.046 7f6e787156d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.046 7f6e787156d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730947230045841 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:40:30.046 7f6e787156d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.084 7f6e81082cc8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:40:30.090 7f6e81082cc8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:40:30.090 7f6e786656d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.091 7f6e786656d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:40:30.091 7f6e786656d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.091 7f6e786656d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.091 7f6e786656d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:40:30.091 7f6e786656d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.106 7f6e81082cc8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:40:30.106 7f6e7865f048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.106 7f6e7865f048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:40:30.106 7f6e7865f048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.114 7f6e81082cc8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:40:30.115 7f6e785a12e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.115 7f6e785a12e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:40:30.115 7f6e785a12e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.132 7f6e81082cc8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:40:30.132 7f6e785a1978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.132 7f6e785a1978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:40:30.133 7f6e785a1978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.147 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:40:30.147 7f6e81082cc8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:40:30.148 7f6e81082cc8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:40:30.148 7f6e81082cc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:40:30.148 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:40:30.149 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:40:30.149 7f6e7859fa20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:40:30.149 7f6e7859fa20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:40:30.150 7f6e785a1d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.150 7f6e785a1d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:40:30.151 7f6e785a1d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.152 7f6e81082cc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:40:30.173 7f6e7e002780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:40:30.173 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:40:30.177 7f6e7859fa20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:40:30.211 7f6e81082cc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:40:30.213 7f6e784a08d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.213 7f6e784a08d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:40:30.213 7f6e784a08d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.219 7f6e81082cc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:40:30.220 7f6e784a3eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.220 7f6e784a3eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:40:30.220 7f6e784a3eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.235 7f6e81082cc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:40:30.287 7f6e81082cc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:40:30.288 7f6e784e2630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.288 7f6e784e2630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:40:30.288 7f6e784e2630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.300 7f6e784e2e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.301 7f6e784e2e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:40:30.301 7f6e784e2e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.309 7f6e784eee10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.309 7f6e784eee10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:40:30.309 7f6e784eee10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:30.310 7f6e81082cc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:40:30.310 7f6e81082cc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:40:30.310 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:40:30.312 7f6e81082cc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:40:30.327 7f6e7e002780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:40:30.335 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:40:30.343 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:40:30.362 7f6e7e002780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:40:30.370 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:40:30.827 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:40:31.012 7f6e81082cc8 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:40:31.196 7f6e81082cc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:40:31.196 7f6e81082cc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:40:31.373 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:31.381 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:40:31.400 7f6e7e002780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:40:31.412 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:40:31.697 7f6e81082cc8 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:40:31.698 7f6e81082cc8 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:40:31.698 7f6e81082cc8 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:40:31.701 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:40:31.702 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:40:31.703 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:40:31.703 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:40:31.704 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:40:31.704 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:40:31.708 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:40:31.708 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:40:31.708 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:40:31.709 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:40:31.709 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:40:31.709 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:40:31.709 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:40:31.709 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:40:31.709 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:40:31.709 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:40:31.709 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:40:31.710 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:40:31.710 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:40:31.711 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:40:31.711 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:40:31.711 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:40:31.711 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:40:31.712 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:40:31.712 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:40:31.712 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:40:31.713 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:40:31.713 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:40:31.713 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:40:31.713 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:40:31.713 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:40:31.713 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:40:31.715 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:40:31.715 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:40:31.715 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:40:31.715 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:40:31.716 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:40:31.716 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:40:31.716 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:40:31.716 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:40:31.716 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:40:31.716 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:40:31.716 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:40:31.716 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:40:31.716 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:40:31.717 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:40:31.717 7f6e81082cc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:40:31.719 7f6e81082cc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:40:31.765 7f6e7828e6d8 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:40:31.766 7f6e7828e6d8 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:40:31.766 7f6e7807f6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:31.766 7f6e7807f6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:40:31.766 7f6e7807f6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:31.787 7f6e7805c780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:31.787 7f6e7805c780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:40:31.787 7f6e7805c780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:40:32.415 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:32.425 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:40:32.447 7f6e7e002780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:40:32.469 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:40:33.473 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:34.045 7f6e72a2eeb8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:40:34.059 7f6e72a2eeb8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:40:34.059 7f6e72a2eeb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:40:34.059 7f6e72a2eeb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:40:34.059 7f6e72a2eeb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:40:34.059 7f6e72a2eeb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:40:34.059 7f6e72a2eeb8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:40:34.060 7f6e72a2eeb8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:40:34.060 7f6e72a2eeb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:40:34.060 7f6e72a2eeb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:40:34.060 7f6e72a2eeb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:40:34.060 7f6e72a456d8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:40:34.062 7f6e72a456d8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:40:34.064 7f6e72a45588 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:40:34.064 7f6e72a45588 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:40:34.065 7f6e792c7e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:40:34.065 7f6e792c7e10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:40:34.065 7f6e792c7e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:40:34.065 7f6e792c7e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:40:34.066 7f6e792c7e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:40:34.066 7f6e72a452e8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:40:34.067 7f6e72a452e8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:40:34.474 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:35.475 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:36.476 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:37.477 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:38.478 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:39.479 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:40.480 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:41.481 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:42.483 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:43.484 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:44.486 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:45.488 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:46.489 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:47.490 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:48.491 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:49.492 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:50.493 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:51.494 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:52.495 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:53.496 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:54.497 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:55.498 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:56.499 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:57.500 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:58.501 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:40:59.503 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:00.504 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:01.506 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:02.507 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:03.508 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:04.509 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:05.510 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:06.511 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:07.512 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:08.513 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:09.514 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:10.516 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:11.517 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:12.519 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:13.520 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:14.521 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:15.521 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:16.523 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:17.524 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:18.525 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:19.527 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:20.528 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:21.529 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:22.530 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:23.532 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:24.284 7f6e72abbeb8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:41:24.533 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:25.533 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:26.535 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:27.536 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:28.537 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:29.538 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:29.859 7f6e72a390f0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:41:30.539 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:31.540 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:32.541 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:33.542 7f6e7e002780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:41:33.583 7f6e81082cc8 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:41:33.735 7f6e81082cc8 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:42:18.033 7f53e4109c88 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:42:18.033 7f53e4109c88 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:42:18.033 7f53e4109c88 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:42:18.033 7f53db7ac6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:42:18.033 7f53db7ac6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:42:18.034 7f53db7ac6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:42:18.034 7f53db7ac6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:42:18.034 7f53db7ac6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.035 7f53db7ac6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:42:18.035 7f53db7ac6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.035 7f53db7ac6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.035 7f53db7ac6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730947338034286 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:42:18.035 7f53db7ac6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.072 7f53e4109c88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:42:18.077 7f53e4109c88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:42:18.078 7f53db7006d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.078 7f53db7006d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:42:18.078 7f53db7006d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.079 7f53db7006d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.079 7f53db7006d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:42:18.079 7f53db7006d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.093 7f53e4109c88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:42:18.093 7f53db6f9048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.094 7f53db6f9048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:42:18.094 7f53db6f9048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.102 7f53e4109c88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:42:18.103 7f53db6392e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.103 7f53db6392e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:42:18.104 7f53db6392e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.124 7f53e4109c88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:42:18.124 7f53db639978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.125 7f53db639978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:42:18.126 7f53db639978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.132 7f53e4109c88 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:42:18.133 7f53e4109c88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:42:18.133 7f53e4109c88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:42:18.134 7f53e4109c88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:42:18.134 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:42:18.134 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:42:18.135 7f53db63aa20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:42:18.135 7f53db63aa20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:42:18.135 7f53db639d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.136 7f53db639d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:42:18.136 7f53db639d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.137 7f53e4109c88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:42:18.152 7f53db63aa20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:42:18.152 7f53e109e780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:42:18.152 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:42:18.195 7f53e4109c88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:42:18.197 7f53db536d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.197 7f53db536d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:42:18.197 7f53db536d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.202 7f53e4109c88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:42:18.202 7f53db58b978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.202 7f53db58b978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:42:18.202 7f53db58b978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.220 7f53e4109c88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:42:18.264 7f53e4109c88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:42:18.264 7f53db53e0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.265 7f53db53e0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:42:18.265 7f53db53e0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.276 7f53db53e8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.277 7f53db53e8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:42:18.277 7f53db53e8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.285 7f53db5848d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.285 7f53db5848d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:42:18.285 7f53db5848d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:18.286 7f53e4109c88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:42:18.286 7f53e4109c88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:42:18.286 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:42:18.286 7f53e4109c88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:42:18.299 7f53e109e780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:42:18.308 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:42:18.315 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:42:18.332 7f53e109e780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:42:18.341 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:42:18.802 7f53e4109c88 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:42:18.966 7f53e4109c88 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:42:19.141 7f53e4109c88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:42:19.142 7f53e4109c88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:42:19.344 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:19.352 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:42:19.368 7f53e109e780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:42:19.379 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:42:19.643 7f53e4109c88 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:42:19.643 7f53e4109c88 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:42:19.643 7f53e4109c88 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:42:19.647 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:42:19.647 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:42:19.648 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:42:19.648 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:42:19.649 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:42:19.649 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:42:19.654 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:42:19.654 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:42:19.654 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:42:19.654 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:42:19.654 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:42:19.654 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:42:19.654 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:42:19.654 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:42:19.654 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:42:19.654 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:42:19.655 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:42:19.655 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:42:19.655 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:42:19.656 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:42:19.656 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:42:19.656 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:42:19.657 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:42:19.657 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:42:19.657 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:42:19.657 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:42:19.658 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:42:19.658 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:42:19.658 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:42:19.658 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:42:19.658 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:42:19.658 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:42:19.660 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:42:19.660 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:42:19.660 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:42:19.660 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:42:19.660 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:42:19.660 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:42:19.660 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:42:19.660 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:42:19.660 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:42:19.661 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:42:19.661 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:42:19.661 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:42:19.661 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:42:19.661 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:42:19.661 7f53e4109c88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:42:19.664 7f53e4109c88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:42:19.712 7f53db3f7f60 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:42:19.712 7f53db3f7f60 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:42:19.713 7f53dabb4198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:19.713 7f53dabb4198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:42:19.713 7f53dabb4198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:19.734 7f53dac0a240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:19.734 7f53dac0a240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:42:19.734 7f53dac0a240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:42:20.383 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:20.392 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:42:20.406 7f53e109e780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:42:20.422 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:42:21.426 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:22.034 7f53daad96d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:42:22.041 7f53daad96d8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:42:22.041 7f53daad96d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:42:22.041 7f53daad96d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:42:22.042 7f53daad96d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:42:22.042 7f53daad96d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:42:22.042 7f53daad96d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:42:22.042 7f53daad96d8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:42:22.043 7f53daad96d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:42:22.043 7f53daad96d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:42:22.043 7f53daad96d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:42:22.043 7f53daaf1198 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:42:22.045 7f53daaf1198 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:42:22.049 7f53daaf1048 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:42:22.049 7f53daaf1048 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:42:22.050 7f53dc361e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:42:22.050 7f53dc361e10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:42:22.050 7f53dc361e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:42:22.050 7f53dc361e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:42:22.050 7f53dc361e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:42:22.051 7f53daad96d8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:42:22.052 7f53daad96d8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:42:22.427 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:23.429 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:24.430 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:25.431 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:26.432 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:27.434 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:28.434 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:29.436 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:30.437 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:31.438 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:32.440 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:33.441 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:34.442 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:35.443 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:36.445 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:37.446 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:38.447 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:39.448 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:40.449 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:41.450 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:42.451 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:43.453 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:44.454 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:45.455 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:46.456 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:47.457 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:48.457 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:49.458 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:50.460 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:51.460 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:52.461 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:53.463 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:54.464 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:55.465 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:56.466 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:57.467 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:58.469 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:42:59.470 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:00.472 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:01.473 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:02.474 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:03.476 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:04.478 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:05.479 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:06.480 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:07.481 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:08.482 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:09.483 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:10.485 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:11.486 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:12.487 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:13.488 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:14.489 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:15.490 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:16.492 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:17.493 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:18.494 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:19.496 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:20.498 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:43:21.263 7f53e4109c88 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:43:21.423 7f53e4109c88 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:43:21.498 7f53e109e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:04.025 7f12297fff88 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:44:04.026 7f12297fff88 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:44:04.026 7f12297fff88 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:44:04.026 7f1220eeb6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:44:04.026 7f1220eeb6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:44:04.027 7f1220eeb6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:44:04.027 7f1220eeb6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:44:04.027 7f1220eeb6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.027 7f1220eeb6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:44:04.027 7f1220eeb6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.027 7f1220eeb6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.028 7f1220eeb6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730947444027043 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:44:04.028 7f1220eeb6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.065 7f12297fff88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:44:04.071 7f12297fff88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:44:04.072 7f1220e3b6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.072 7f1220e3b6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:44:04.072 7f1220e3b6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.072 7f1220e3b6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.072 7f1220e3b6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:44:04.072 7f1220e3b6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.085 7f12297fff88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:44:04.086 7f1220e34048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.086 7f1220e34048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:44:04.086 7f1220e34048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.094 7f12297fff88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:44:04.095 7f1220d772e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.095 7f1220d772e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:44:04.095 7f1220d772e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.109 7f12297fff88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:44:04.110 7f1220d77780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.110 7f1220d77780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:44:04.110 7f1220d77780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.119 7f12297fff88 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:44:04.119 7f12297fff88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:44:04.119 7f12297fff88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:44:04.120 7f12297fff88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:44:04.120 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:44:04.120 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:44:04.121 7f1229728438 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:44:04.121 7f1229728438 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:44:04.121 7f1220d77d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.121 7f1220d77d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:44:04.121 7f1220d77d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.123 7f12297fff88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:44:04.142 7f1229728438 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:44:04.144 7f1220d72f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:44:04.145 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:44:04.181 7f12297fff88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:44:04.183 7f1220c73d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.183 7f1220c73d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:44:04.183 7f1220c73d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.189 7f12297fff88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:44:04.190 7f1220cc4978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.190 7f1220cc4978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:44:04.190 7f1220cc4978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.202 7f12297fff88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:44:04.245 7f12297fff88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:44:04.246 7f1220c620f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.246 7f1220c620f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:44:04.246 7f1220c620f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.258 7f1220c628d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.259 7f1220c628d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:44:04.259 7f1220c628d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.268 7f1220cc38d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.268 7f1220cc38d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:44:04.269 7f1220cc38d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:04.269 7f12297fff88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:44:04.269 7f12297fff88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:44:04.269 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:44:04.270 7f12297fff88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:44:04.283 7f1220d72f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:44:04.292 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:44:04.300 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:44:04.313 7f1220d72f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:44:04.321 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:44:04.783 7f12297fff88 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:44:04.947 7f12297fff88 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:44:05.153 7f12297fff88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:44:05.153 7f12297fff88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:44:05.323 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:05.338 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:44:05.354 7f1220d72f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:44:05.368 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:44:05.654 7f12297fff88 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:44:05.654 7f12297fff88 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:44:05.655 7f12297fff88 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:44:05.658 7f12297fff88 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:44:05.659 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:44:05.660 7f12297fff88 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:44:05.660 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:44:05.661 7f12297fff88 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:44:05.661 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:44:05.667 7f12297fff88 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:44:05.667 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:44:05.667 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:44:05.667 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:44:05.667 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:44:05.667 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:44:05.667 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:44:05.667 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:44:05.667 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:44:05.668 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:44:05.668 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:44:05.668 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:44:05.668 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:44:05.670 7f12297fff88 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:44:05.670 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:44:05.670 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:44:05.670 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:44:05.671 7f12297fff88 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:44:05.671 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:44:05.671 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:44:05.671 7f12297fff88 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:44:05.671 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:44:05.671 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:44:05.672 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:44:05.672 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:44:05.672 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:44:05.673 7f12297fff88 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:44:05.673 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:44:05.673 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:44:05.673 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:44:05.673 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:44:05.673 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:44:05.673 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:44:05.673 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:44:05.674 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:44:05.674 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:44:05.674 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:44:05.674 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:44:05.674 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:44:05.674 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:44:05.674 7f12297fff88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:44:05.677 7f12297fff88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:44:05.725 7f1220b32780 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:44:05.725 7f1220b32780 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:44:05.726 7f12202f0198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:05.726 7f12202f0198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:44:05.727 7f12202f0198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:05.747 7f1220345240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:05.747 7f1220345240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:44:05.747 7f1220345240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:44:06.372 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:06.384 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:44:06.410 7f1220d72f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:44:06.424 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:44:07.428 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:08.027 7f12202156d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:44:08.034 7f12202156d8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:44:08.034 7f12202156d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:44:08.035 7f12202156d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:44:08.035 7f12202156d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:44:08.035 7f12202156d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:44:08.035 7f12202156d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:44:08.035 7f12202156d8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:44:08.036 7f12202156d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:44:08.036 7f12202156d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:44:08.036 7f12202156d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:44:08.036 7f122022c198 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:44:08.037 7f122022c198 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:44:08.040 7f122022c048 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:44:08.040 7f122022c048 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:44:08.040 7f1221b06d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:44:08.041 7f1221b06d68 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:44:08.041 7f1221b06d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:44:08.041 7f1221b06d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:44:08.041 7f1221b06d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:44:08.041 7f12202156d8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:44:08.043 7f12202156d8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:44:08.430 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:09.431 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:10.432 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:11.433 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:12.434 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:13.435 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:14.437 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:15.438 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:16.440 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:17.440 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:18.441 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:19.442 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:20.443 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:21.444 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:22.445 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:23.447 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:24.448 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:25.449 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:26.450 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:27.451 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:28.452 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:29.453 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:30.454 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:31.455 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:32.456 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:33.457 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:34.459 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:35.460 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:36.461 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:37.462 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:38.463 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:39.464 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:40.465 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:41.466 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:42.468 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:43.469 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:44.470 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:45.471 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:46.472 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:47.473 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:48.474 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:49.475 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:50.476 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:51.477 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:52.284 7f122021bcc0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:44:52.479 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:53.480 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:54.481 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:55.483 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:56.484 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:57.485 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:58.486 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:44:59.488 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:00.490 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:01.491 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:02.492 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:03.493 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:04.494 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:05.495 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:06.497 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:07.027 7f12297fff88 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:45:07.179 7f12297fff88 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:45:07.636 7f1220d72f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:50.036 7f6fd0659bc8 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:45:50.037 7f6fd0659bc8 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:45:50.037 7f6fd0659bc8 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:45:50.037 7f6fc3cb26d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:45:50.037 7f6fc3cb26d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:45:50.038 7f6fc3cb26d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:45:50.038 7f6fc3cb26d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:45:50.038 7f6fc3cb26d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.038 7f6fc3cb26d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:45:50.038 7f6fc3cb26d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.038 7f6fc3cb26d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.038 7f6fc3cb26d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730947550038122 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:45:50.039 7f6fc3cb26d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.077 7f6fd0659bc8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:45:50.082 7f6fd0659bc8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:45:50.083 7f6fc3c066d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.083 7f6fc3c066d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:45:50.083 7f6fc3c066d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.083 7f6fc3c066d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.083 7f6fc3c066d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:45:50.083 7f6fc3c066d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.100 7f6fd0659bc8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:45:50.101 7f6fc3bff048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.101 7f6fc3bff048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:45:50.101 7f6fc3bff048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.110 7f6fd0659bc8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:45:50.111 7f6fc3b412e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.111 7f6fc3b412e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:45:50.111 7f6fc3b412e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.126 7f6fd0659bc8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:45:50.126 7f6fc3b41978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.126 7f6fc3b41978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:45:50.126 7f6fc3b41978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.136 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:45:50.137 7f6fd0659bc8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:45:50.137 7f6fd0659bc8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:45:50.137 7f6fd0659bc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:45:50.138 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:45:50.138 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:45:50.138 7f6fc3b3fa20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:45:50.138 7f6fc3b3fa20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:45:50.139 7f6fc3b41d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.139 7f6fc3b41d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:45:50.139 7f6fc3b41d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.140 7f6fd0659bc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:45:50.155 7f6fc3b3fa20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:45:50.156 7f6fcd5ac780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:45:50.156 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:45:50.203 7f6fd0659bc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:45:50.205 7f6fc3a3cd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.205 7f6fc3a3cd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:45:50.205 7f6fc3a3cd68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.212 7f6fd0659bc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:45:50.213 7f6fc3a8e978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.213 7f6fc3a8e978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:45:50.213 7f6fc3a8e978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.227 7f6fd0659bc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:45:50.264 7f6fd0659bc8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:45:50.265 7f6fc3a7f0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.265 7f6fc3a7f0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:45:50.265 7f6fc3a7f0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.277 7f6fc3a7f8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.278 7f6fc3a7f8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:45:50.278 7f6fc3a7f8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.287 7f6fc3a8d8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.287 7f6fc3a8d8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:45:50.287 7f6fc3a8d8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:50.288 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:45:50.288 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:45:50.288 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:45:50.288 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:45:50.313 7f6fcd5ac780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:45:50.324 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:45:50.332 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:45:50.349 7f6fcd5ac780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:45:50.356 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:45:50.801 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:45:50.972 7f6fd0659bc8 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:45:51.153 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:45:51.154 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:45:51.359 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:51.367 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:45:51.383 7f6fcd5ac780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:45:51.394 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:45:51.654 7f6fd0659bc8 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:45:51.654 7f6fd0659bc8 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:45:51.655 7f6fd0659bc8 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:45:51.658 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:45:51.658 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:45:51.659 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:45:51.659 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:45:51.661 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:45:51.662 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:45:51.666 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:45:51.666 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:45:51.666 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:45:51.667 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:45:51.667 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:45:51.667 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:45:51.667 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:45:51.667 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:45:51.667 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:45:51.667 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:45:51.667 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:45:51.667 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:45:51.667 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:45:51.669 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:45:51.669 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:45:51.669 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:45:51.669 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:45:51.670 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:45:51.670 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:45:51.670 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:45:51.671 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:45:51.671 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:45:51.671 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:45:51.671 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:45:51.671 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:45:51.671 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:45:51.672 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:45:51.672 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:45:51.672 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:45:51.673 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:45:51.673 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:45:51.673 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:45:51.673 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:45:51.673 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:45:51.673 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:45:51.673 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:45:51.673 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:45:51.673 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:45:51.674 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:45:51.674 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:45:51.675 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:45:51.677 7f6fd0659bc8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:45:51.725 7f6fc38fcac8 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:45:51.725 7f6fc38fcac8 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:45:51.725 7f6fc30b9198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:51.726 7f6fc30b9198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:45:51.726 7f6fc30b9198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:51.747 7f6fc310e240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:51.747 7f6fc310e240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:45:51.747 7f6fc310e240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:45:52.398 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:52.405 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:45:52.419 7f6fcd5ac780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:45:52.440 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:45:53.443 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:54.038 7f6fc2fe06d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:45:54.045 7f6fc2fe06d8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:45:54.045 7f6fc2fe06d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:45:54.045 7f6fc2fe06d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:45:54.045 7f6fc2fe06d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:45:54.046 7f6fc2fe06d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:45:54.046 7f6fc2fe06d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:45:54.046 7f6fc2fe06d8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:45:54.046 7f6fc2fe06d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:45:54.046 7f6fc2fe06d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:45:54.046 7f6fc2fe06d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:45:54.047 7f6fc2ff7198 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:45:54.048 7f6fc2ff7198 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:45:54.057 7f6fc2ff7048 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:45:54.057 7f6fc2ff7048 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:45:54.058 7f6fc886ee10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:45:54.058 7f6fc886ee10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:45:54.058 7f6fc886ee10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:45:54.058 7f6fc886ee10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:45:54.058 7f6fc886ee10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:45:54.059 7f6fc2fe0f60 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:45:54.060 7f6fc2fe0f60 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:45:54.445 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:55.445 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:56.447 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:57.448 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:58.449 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:45:59.451 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:00.452 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:01.454 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:02.454 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:03.455 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:04.456 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:05.457 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:06.457 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:07.459 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:08.460 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:09.461 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:10.462 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:11.464 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:12.465 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:13.466 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:14.467 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:15.468 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:16.469 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:17.471 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:18.472 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:19.473 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:20.474 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:21.475 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:22.476 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:23.477 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:24.478 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:25.480 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:26.480 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:27.482 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:28.483 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:29.485 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:30.486 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:31.487 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:32.488 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:33.489 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:34.490 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:35.492 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:36.493 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:37.493 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:38.495 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:39.496 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:40.497 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:41.499 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:42.500 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:43.502 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:44.502 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:45.504 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:46.506 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:47.507 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:48.508 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:49.509 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:50.511 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:51.512 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:52.513 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:46:52.927 7f6fd0659bc8 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:46:53.079 7f6fd0659bc8 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:46:53.525 7f6fcd5ac780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:38.016 7efdb453ff08 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:47:38.016 7efdb453ff08 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:47:38.016 7efdb453ff08 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:47:38.016 7efdabb6d6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:47:38.017 7efdabb6d6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:47:38.017 7efdabb6d6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:47:38.017 7efdabb6d6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:47:38.018 7efdabb6d6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.019 7efdabb6d6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:47:38.019 7efdabb6d6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.019 7efdabb6d6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.019 7efdabb6d6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730947658017456 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:47:38.019 7efdabb6d6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.054 7efdb453ff08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:47:38.059 7efdb453ff08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:47:38.060 7efdababd6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.060 7efdababd6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:47:38.060 7efdababd6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.060 7efdababd6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.060 7efdababd6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:47:38.060 7efdababd6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.073 7efdb453ff08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:47:38.073 7efdabab7048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.074 7efdabab7048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:47:38.074 7efdabab7048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.082 7efdb453ff08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:47:38.083 7efdab9f72e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.083 7efdab9f72e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:47:38.083 7efdab9f72e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.096 7efdb453ff08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:47:38.097 7efdab9f7780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.097 7efdab9f7780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:47:38.097 7efdab9f7780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.105 7efdb453ff08 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:47:38.106 7efdb453ff08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:47:38.106 7efdb453ff08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:47:38.106 7efdb453ff08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:47:38.107 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:47:38.107 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:47:38.107 7efdb43a1390 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:47:38.107 7efdb43a1390 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:47:38.108 7efdab9f7d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.108 7efdab9f7d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:47:38.108 7efdab9f7d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.110 7efdb453ff08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:47:38.134 7efdb43a1390 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:47:38.135 7efdab9f9f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:47:38.135 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:47:38.166 7efdb453ff08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:47:38.168 7efdab8f4d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.168 7efdab8f4d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:47:38.168 7efdab8f4d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.173 7efdb453ff08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:47:38.173 7efdab946978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.173 7efdab946978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:47:38.173 7efdab946978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.187 7efdb453ff08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:47:38.229 7efdb453ff08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:47:38.230 7efdab8fa0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.230 7efdab8fa0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:47:38.230 7efdab8fa0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.245 7efdab8fa8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.245 7efdab8fa8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:47:38.245 7efdab8fa8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.254 7efdab9418d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.254 7efdab9418d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:47:38.254 7efdab9418d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:38.254 7efdb453ff08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:47:38.254 7efdb453ff08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:47:38.255 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:47:38.255 7efdb453ff08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:47:38.268 7efdab9f9f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:47:38.276 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:47:38.284 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:47:38.297 7efdab9f9f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:47:38.305 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:47:38.768 7efdb453ff08 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:47:38.947 7efdb453ff08 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:47:39.125 7efdb453ff08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:47:39.125 7efdb453ff08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:47:39.307 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:39.314 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:47:39.332 7efdab9f9f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:47:39.348 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:47:39.626 7efdb453ff08 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:47:39.627 7efdb453ff08 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:47:39.627 7efdb453ff08 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:47:39.631 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:47:39.631 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:47:39.632 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:47:39.632 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:47:39.633 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:47:39.633 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:47:39.639 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:47:39.639 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:47:39.639 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:47:39.639 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:47:39.639 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:47:39.639 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:47:39.639 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:47:39.640 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:47:39.640 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:47:39.641 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:47:39.641 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:47:39.641 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:47:39.641 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:47:39.643 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:47:39.643 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:47:39.643 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:47:39.643 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:47:39.644 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:47:39.644 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:47:39.645 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:47:39.645 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:47:39.646 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:47:39.646 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:47:39.646 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:47:39.646 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:47:39.646 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:47:39.647 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:47:39.647 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:47:39.647 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:47:39.648 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:47:39.648 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:47:39.648 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:47:39.648 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:47:39.648 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:47:39.648 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:47:39.649 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:47:39.649 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:47:39.650 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:47:39.650 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:47:39.650 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:47:39.650 7efdb453ff08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:47:39.653 7efdb453ff08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:47:39.698 7efdab7b4ac8 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:47:39.699 7efdab7b4ac8 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:47:39.699 7efdaaf72198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:39.699 7efdaaf72198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:47:39.699 7efdaaf72198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:39.720 7efdaafc6240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:39.720 7efdaafc6240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:47:39.721 7efdaafc6240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:47:40.352 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:40.364 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:47:40.376 7efdab9f9f60 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:47:40.389 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:47:41.392 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:42.017 7efdaa697c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:47:42.026 7efdaa697c18 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:47:42.027 7efdaa697c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:47:42.027 7efdaa697c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:47:42.027 7efdaa697c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:47:42.027 7efdaa697c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:47:42.027 7efdaa697c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:47:42.027 7efdaa697c18 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:47:42.028 7efdaa697c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:47:42.028 7efdaa697c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:47:42.028 7efdaa697c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:47:42.029 7efdaa6ac438 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:47:42.030 7efdaa6ac438 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:47:42.032 7efdaa6ac2e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:47:42.033 7efdaa6ac2e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:47:42.033 7efdac788d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:47:42.033 7efdac788d68 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:47:42.033 7efdac788d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:47:42.034 7efdac788d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:47:42.034 7efdac788d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:47:42.034 7efdaa6ac048 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:47:42.035 7efdaa6ac048 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:47:42.393 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:43.393 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:44.394 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:45.395 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:46.396 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:47.397 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:48.398 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:49.399 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:50.401 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:51.401 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:52.402 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:53.403 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:54.405 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:55.405 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:56.407 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:57.408 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:58.409 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:47:59.410 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:00.411 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:01.412 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:02.413 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:03.414 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:04.417 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:05.418 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:06.421 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:07.422 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:08.423 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:09.424 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:10.425 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:11.427 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:12.428 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:13.430 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:14.431 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:15.432 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:16.433 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:17.434 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:18.435 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:19.436 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:20.437 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:21.438 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:22.440 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:23.441 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:24.442 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:25.443 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:26.444 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:27.445 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:28.446 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:29.448 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:30.449 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:31.451 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:32.452 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:33.453 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:34.454 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:35.456 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:36.457 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:37.459 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:38.460 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:39.461 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:40.461 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:48:41.079 7efdb453ff08 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:48:41.243 7efdb453ff08 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:48:41.462 7efdab9f9f60 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:24.063 7fa97233cd48 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:49:24.064 7fa97233cd48 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:49:24.064 7fa97233cd48 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:49:24.064 7fa969a256d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:49:24.064 7fa969a256d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:49:24.065 7fa969a256d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:49:24.065 7fa969a256d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:49:24.065 7fa969a256d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.065 7fa969a256d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:49:24.065 7fa969a256d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.065 7fa969a256d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.066 7fa969a256d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730947764065228 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:49:24.066 7fa969a256d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.103 7fa97233cd48 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:49:24.108 7fa97233cd48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:49:24.108 7fa9699776d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.109 7fa9699776d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:49:24.109 7fa9699776d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.109 7fa9699776d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.109 7fa9699776d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:49:24.109 7fa9699776d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.123 7fa97233cd48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:49:24.123 7fa969970048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.124 7fa969970048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:49:24.124 7fa969970048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.130 7fa97233cd48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:49:24.131 7fa9698b32e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.131 7fa9698b32e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:49:24.131 7fa9698b32e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.143 7fa97233cd48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:49:24.144 7fa9698b3978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.144 7fa9698b3978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:49:24.144 7fa9698b3978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.152 7fa97233cd48 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:49:24.153 7fa97233cd48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:49:24.153 7fa97233cd48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:49:24.153 7fa97233cd48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:49:24.154 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:49:24.154 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:49:24.154 7fa9698b1a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:49:24.155 7fa9698b1a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:49:24.155 7fa9698b3d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.155 7fa9698b3d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:49:24.155 7fa9698b3d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.157 7fa97233cd48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:49:24.175 7fa96f316780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:49:24.175 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:49:24.176 7fa9698b1a20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:49:24.216 7fa97233cd48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:49:24.218 7fa9697aed68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.218 7fa9697aed68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:49:24.218 7fa9697aed68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.223 7fa97233cd48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:49:24.223 7fa969806978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.223 7fa969806978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:49:24.223 7fa969806978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.236 7fa97233cd48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:49:24.277 7fa97233cd48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:49:24.277 7fa9697f20f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.278 7fa9697f20f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:49:24.278 7fa9697f20f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.290 7fa9697f28d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.290 7fa9697f28d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:49:24.290 7fa9697f28d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.298 7fa9697ff8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.299 7fa9697ff8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:49:24.299 7fa9697ff8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:24.299 7fa97233cd48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:49:24.299 7fa97233cd48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:49:24.299 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:49:24.300 7fa97233cd48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:49:24.314 7fa96f316780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:49:24.322 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:49:24.330 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:49:24.345 7fa96f316780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:49:24.353 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:49:24.814 7fa97233cd48 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:49:25.008 7fa97233cd48 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:49:25.214 7fa97233cd48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:49:25.214 7fa97233cd48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:49:25.356 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:25.362 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:49:25.379 7fa96f316780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:49:25.392 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:49:25.715 7fa97233cd48 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:49:25.715 7fa97233cd48 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:49:25.715 7fa97233cd48 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:49:25.719 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:49:25.719 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:49:25.720 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:49:25.720 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:49:25.721 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:49:25.721 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:49:25.726 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:49:25.727 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:49:25.727 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:49:25.727 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:49:25.727 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:49:25.727 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:49:25.727 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:49:25.727 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:49:25.727 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:49:25.727 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:49:25.728 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:49:25.728 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:49:25.728 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:49:25.729 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:49:25.729 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:49:25.730 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:49:25.730 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:49:25.730 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:49:25.730 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:49:25.731 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:49:25.731 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:49:25.731 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:49:25.731 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:49:25.731 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:49:25.731 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:49:25.732 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:49:25.733 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:49:25.733 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:49:25.733 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:49:25.733 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:49:25.733 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:49:25.733 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:49:25.733 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:49:25.733 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:49:25.733 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:49:25.734 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:49:25.734 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:49:25.734 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:49:25.734 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:49:25.734 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:49:25.734 7fa97233cd48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:49:25.737 7fa97233cd48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:49:25.785 7fa96966feb8 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:49:25.786 7fa96966feb8 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:49:25.786 7fa968e2c198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:25.786 7fa968e2c198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:49:25.786 7fa968e2c198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:25.808 7fa968e7f240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:25.809 7fa968e7f240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:49:25.809 7fa968e7f240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:49:26.396 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:26.408 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:49:26.423 7fa96f316780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:49:26.446 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:49:27.451 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:28.065 7fa968d516d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:49:28.072 7fa968d516d8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:49:28.073 7fa968d516d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:49:28.073 7fa968d516d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:49:28.073 7fa968d516d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:49:28.073 7fa968d516d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:49:28.073 7fa968d516d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:49:28.073 7fa968d516d8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:49:28.074 7fa968d516d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:49:28.074 7fa968d516d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:49:28.074 7fa968d516d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:49:28.074 7fa968d8a630 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:49:28.075 7fa968d8a630 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:49:28.078 7fa968d51eb8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:49:28.079 7fa968d51eb8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:49:28.079 7fa96a5d8e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:49:28.079 7fa96a5d8e10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:49:28.080 7fa96a5d8e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:49:28.080 7fa96a5d8e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:49:28.080 7fa96a5d8e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:49:28.080 7fa968d8a588 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:49:28.081 7fa968d8a588 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:49:28.453 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:29.454 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:30.456 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:31.457 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:32.458 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:33.459 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:34.461 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:35.462 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:36.463 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:37.464 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:38.465 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:39.467 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:40.468 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:41.470 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:42.471 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:42.826 7fa969779198 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:49:43.472 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:44.473 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:45.474 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:46.475 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:47.476 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:48.478 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:49.479 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:50.479 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:51.480 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:52.482 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:53.482 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:54.483 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:55.484 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:56.486 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:57.487 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:58.489 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:49:59.490 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:00.493 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:01.494 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:02.495 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:03.496 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:04.497 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:05.498 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:06.499 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:07.500 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:08.501 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:09.502 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:10.503 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:11.505 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:12.506 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:13.507 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:14.508 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:15.510 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:16.511 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:17.513 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:18.514 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:19.515 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:20.516 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:21.517 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:21.922 7fa96f47fa20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:50:22.519 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:23.520 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:24.522 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:25.523 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:26.523 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:27.524 7fa96f316780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:50:27.607 7fa97233cd48 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:50:27.775 7fa97233cd48 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:51:10.003 7fcbce41a6c8 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/07/2024 02:51:10.003 7fcbce41a6c8 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/07/2024 02:51:10.003 7fcbce41a6c8 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/07/2024 02:51:10.003 7fcbc5aeb6d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 11/07/2024 02:51:10.004 7fcbc5aeb6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:51:10.004 7fcbc5aeb6d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:51:10.004 7fcbc5aeb6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/07/2024 02:51:10.004 7fcbc5aeb6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.004 7fcbc5aeb6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:51:10.004 7fcbc5aeb6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.004 7fcbc5aeb6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.005 7fcbc5aeb6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-444-1:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 110 start_time = 1730947870004297 collector_name = collector_ip = collector_list = [ 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:51:10.005 7fcbc5aeb6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.041 7fcbce41a6c8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.49:5673//'] 11/07/2024 02:51:10.046 7fcbce41a6c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:51:10.047 7fcbc5a3a6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.047 7fcbc5a3a6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:51:10.047 7fcbc5a3a6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.047 7fcbc5a3a6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.047 7fcbc5a3a6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:51:10.047 7fcbc5a3a6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.066 7fcbce41a6c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:51:10.066 7fcbc5a34048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.067 7fcbc5a34048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:51:10.067 7fcbc5a34048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.075 7fcbce41a6c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:51:10.075 7fcbc59772e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.076 7fcbc59772e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:51:10.076 7fcbc59772e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.089 7fcbce41a6c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:51:10.090 7fcbc5977978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.091 7fcbc5977978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:51:10.091 7fcbc5977978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.101 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 11/07/2024 02:51:10.101 7fcbce41a6c8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:51:10.102 7fcbce41a6c8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 11/07/2024 02:51:10.102 7fcbce41a6c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:51:10.103 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 11/07/2024 02:51:10.103 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:51:10.103 7fcbc5975a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 11/07/2024 02:51:10.103 7fcbc5975a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 11/07/2024 02:51:10.103 7fcbc5977d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.104 7fcbc5977d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:51:10.104 7fcbc5977d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.105 7fcbce41a6c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:51:10.125 7fcbc5975a20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 11/07/2024 02:51:10.129 7fcbcb3da780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:51:10.129 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 11/07/2024 02:51:10.164 7fcbce41a6c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:51:10.165 7fcbc5872d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.166 7fcbc5872d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:51:10.166 7fcbc5872d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.170 7fcbce41a6c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:51:10.171 7fcbc58cc978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.171 7fcbc58cc978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:51:10.171 7fcbc58cc978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.183 7fcbce41a6c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:51:10.228 7fcbce41a6c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:51:10.229 7fcbc58b60f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.230 7fcbc58b60f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:51:10.230 7fcbc58b60f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.245 7fcbc58b68d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.245 7fcbc58b68d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:51:10.245 7fcbc58b68d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.253 7fcbc58c48d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.254 7fcbc58c48d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:51:10.254 7fcbc58c48d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:10.254 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 11/07/2024 02:51:10.254 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 11/07/2024 02:51:10.254 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:51:10.257 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1 11/07/2024 02:51:10.269 7fcbcb3da780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:51:10.278 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:51:10.285 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:51:10.300 7fcbcb3da780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:51:10.308 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1'] 11/07/2024 02:51:10.773 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Fetching all containers 11/07/2024 02:51:10.976 7fcbce41a6c8 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_api_dnsmasq_1 11/07/2024 02:51:11.192 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue 11/07/2024 02:51:11.192 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue 11/07/2024 02:51:11.311 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:11.319 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:51:11.336 7fcbcb3da780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:51:11.350 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue'] 11/07/2024 02:51:11.694 7fcbce41a6c8 [contrail-device-manager] [WARNING]: Process 110 prepared to run in Full mode... 11/07/2024 02:51:11.694 7fcbce41a6c8 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 11/07/2024 02:51:11.694 7fcbce41a6c8 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 11/07/2024 02:51:11.698 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 11/07/2024 02:51:11.698 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 11/07/2024 02:51:11.699 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 11/07/2024 02:51:11.699 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 11/07/2024 02:51:11.700 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 11/07/2024 02:51:11.700 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 11/07/2024 02:51:11.707 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 11/07/2024 02:51:11.707 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 11/07/2024 02:51:11.707 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 11/07/2024 02:51:11.707 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 11/07/2024 02:51:11.707 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 11/07/2024 02:51:11.707 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 11/07/2024 02:51:11.708 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 11/07/2024 02:51:11.708 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 11/07/2024 02:51:11.708 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 11/07/2024 02:51:11.708 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 11/07/2024 02:51:11.708 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 11/07/2024 02:51:11.708 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 11/07/2024 02:51:11.709 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 11/07/2024 02:51:11.711 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 11/07/2024 02:51:11.712 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 11/07/2024 02:51:11.712 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 11/07/2024 02:51:11.712 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 11/07/2024 02:51:11.712 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 11/07/2024 02:51:11.712 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 11/07/2024 02:51:11.712 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 11/07/2024 02:51:11.713 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 11/07/2024 02:51:11.713 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 11/07/2024 02:51:11.713 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 11/07/2024 02:51:11.713 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 11/07/2024 02:51:11.713 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 11/07/2024 02:51:11.714 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 11/07/2024 02:51:11.715 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 11/07/2024 02:51:11.715 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 11/07/2024 02:51:11.715 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 11/07/2024 02:51:11.715 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 11/07/2024 02:51:11.715 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 11/07/2024 02:51:11.715 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 11/07/2024 02:51:11.715 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 11/07/2024 02:51:11.715 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 11/07/2024 02:51:11.716 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 11/07/2024 02:51:11.716 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 11/07/2024 02:51:11.716 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 11/07/2024 02:51:11.716 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 11/07/2024 02:51:11.716 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 11/07/2024 02:51:11.716 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 11/07/2024 02:51:11.716 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 11/07/2024 02:51:11.719 7fcbce41a6c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 11/07/2024 02:51:11.768 7fcbc5733390 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 11/07/2024 02:51:11.768 7fcbc5733390 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 11/07/2024 02:51:11.768 7fcbc4eef198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:11.769 7fcbc4eef198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 11/07/2024 02:51:11.769 7fcbc4eef198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:11.793 7fcbc4f44240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:11.794 7fcbc4f44240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:51:11.794 7fcbc4f44240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/07/2024 02:51:12.353 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:12.367 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 11/07/2024 02:51:12.383 7fcbcb3da780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 11/07/2024 02:51:12.403 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-juju-os-444-1', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-juju-os-444-1.tftp_queue', 'device_manager_ztp.ztp_queue'] 11/07/2024 02:51:13.410 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:14.006 7fcbc4e4d8d0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/07/2024 02:51:14.013 7fcbc4e4d8d0 [contrail-device-manager] [INFO]: Session Event: TCP Connected 11/07/2024 02:51:14.013 7fcbc4e4d8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/07/2024 02:51:14.013 7fcbc4e4d8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:51:14.013 7fcbc4e4d8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-444-1 process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.49:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.49:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.49:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.49:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/07/2024 02:51:14.013 7fcbc4e4d8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/07/2024 02:51:14.013 7fcbc4e4d8d0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/07/2024 02:51:14.014 7fcbc4e4d8d0 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 11/07/2024 02:51:14.014 7fcbc4e4d8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/07/2024 02:51:14.014 7fcbc4e4d8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:51:14.014 7fcbc4e4d8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/07/2024 02:51:14.014 7fcbc4e14ac8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:51:14.016 7fcbc4e14ac8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:51:14.018 7fcbc4e14f60 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001012 11/07/2024 02:51:14.019 7fcbc4e14f60 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/07/2024 02:51:14.020 7fcbc4eef198 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/07/2024 02:51:14.020 7fcbc4eef198 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 11/07/2024 02:51:14.020 7fcbc4eef198 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 11/07/2024 02:51:14.020 7fcbc4eef198 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/07/2024 02:51:14.020 7fcbc4eef198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/07/2024 02:51:14.020 7fcbc4e14ac8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/07/2024 02:51:14.023 7fcbc4e14ac8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/07/2024 02:51:14.411 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:15.412 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:16.413 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:17.414 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:18.415 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:19.416 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:20.417 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:21.418 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:22.419 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:23.420 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:24.421 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:25.423 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:26.424 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:27.425 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:28.426 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:29.427 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:30.429 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:31.429 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:32.431 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:33.432 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:34.433 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:35.435 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:36.436 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:37.437 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:38.438 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:39.440 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:40.440 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:41.441 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:42.442 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:43.444 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:44.445 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:45.445 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:46.447 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:47.448 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:48.449 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:49.450 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:50.451 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:51.452 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:52.453 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:53.455 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:54.456 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:55.457 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:56.458 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:57.460 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:58.461 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:51:59.463 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:52:00.464 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:52:01.465 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:52:02.467 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:52:03.468 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:52:04.469 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:52:05.470 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:52:06.470 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:52:07.472 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:52:08.473 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:52:09.474 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:52:10.475 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:52:11.476 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:52:12.477 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 11/07/2024 02:52:13.039 7fcbce41a6c8 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6e0b5675-48c4-4601-8109-29a4fe719109 11/07/2024 02:52:13.199 7fcbce41a6c8 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 148948cb-661d-4b2a-985d-a3daef86734f 11/07/2024 02:52:13.478 7fcbcb3da780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: socket.timeout occurred while draining events, no events to process.